For the commands not listed below, refer to the tables above. RETRY resubmit requests that fail under certain error conditions. BTEQ vs. Yet, BTEQ is not a utility designed for bulk data movement.
|Published (Last):||14 December 2012|
|PDF File Size:||10.9 Mb|
|ePub File Size:||3.43 Mb|
|Price:||Free* [*Free Regsitration Required]|
Although this approach has been implemented and tested internally, it is offered on an as-is basis. Disclaimer: This guide includes content from both Microsoft and Teradata product documentation.
Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation. There are two types of activities that you can use in an Azure Data Factory pipeline. Data movement to move data between supported source and sink data stores. Fortunately, you can create a Custom Activity with your own data movement or transformation logic and use the activity in a pipeline in Data Factory.
The custom activity runs your customized code logic on an Azure Batch pool of virtual machines. This is a diagram of the workflow. You will need to install the Vantage client software on a compute node. Prerequisites You are expected to be familiar with Azure services and Teradata. You will need the following accounts, objects, and systems.
Links have been included with setup instructions. A Microsoft Azure subscription. You can start with a free account. An Azure Blob Storage account and container. You will need to install the Vantage client software on the compute nodes. An Azure Data Factory instance and a pipeline with custom activity. Teradata Vantage with a user and password.
The user must have permission and space to create a table. Vantage must be accessible from Azure services. Sample data file, which will be uploaded to your storage account. The scripts which the Custom Activity will access and execute the load and transformation scripts. We suggest creating the storage account with two 2 containers. Call one of the containers data for the dataset and the other container files for scripts.
This document will use the names data and files for the containers. You call them something entirely different. However, for this document, we will refer to the containers as data and files. You will need to adjust the directions for the names that you actually used. Installs the applications you want to run and schedules jobs to run on the virtual machines. Logon to the Azure portal and create a Batch account.
After deployment, click on Go to resource to view the Batch account in preparation for the next step. Create a pool of compute nodes We need to create a pool of virtual machines, which can be either Windows- or Linux-based. We will use Windows. For simplicity, we will only create a pool with a single virtual machine.
In production, you would create more virtual machines for scaling and redundancy. Enter a Pool ID name. For the Operating System, choose the following. For the Node Size, choose the following.
For the Scale, choose the following. Leave the other options at their default values. Click the Save button. Batch will create the pool immediately, but it will take a few minutes to allocate and start the virtual machines.
You may need to click on Refresh to update the Allocation state. Click on Nodes in the left pane and then click on your specific node. Create a user on the virtual machine by clicking on Connect. Select Specify your own option. Click on Add user account. Click on Download RDP file. Alternatively, you can copy the IP address and port and manually connect with the Remote Desktop application. Note that the password must be sufficiently complex. An error message will remind you of what characters and length are required.
We will use the remote connection to the server to install software in the next section. However, you can use other client software tools and scripting languages, such as Teradata Parallel Transporter or the Teradata Vantage Python package for Advanced Analytics , as an alternative. Log into the Windows virtual machine created in the previous step, if you have not already done so. You can use the downloaded RDP file as a shortcut.
You will need the username and password you created in the previous section. Open Server Manager. Select Local Server in the left pane. While this is normally a security concern, it is acceptable for our purposes. You can always turn it back on. Open Internet Explorer and go to the Teradata Downloads site. If you do not have a login, you will need to register. You will be asked to login and agree to the licensing provisions.
Choose to save the file. The client software is periodically updated. The version will change. The latest version is acceptable. Once completed, open the downloaded file. It appears as a folder as it is a compressed file. Right-click on the file and select Extract all. A new Explorer window will appear. Double-click to run the setup program. Accept any defaults.
The BTEQ program will start. We will put them in the previously created container, files. We will then load a data file from an Azure Blob Storage container, called data, into Vantage.
For our access module load script to access Azure Blob Storage account, we need to create a credentials file. You may place the credentials file where you want as you can use the -ConfigDir parameter to specify this path. We will create a directory, azureaxsmod, in the Teradata directory.
The file name is credentials. Here is a sample credentials file. The StorageAccountKey below is abbreviated. You will need to ensure that your full key in your file. Note that the file name has no extension. Replace any value within the angled brackets with actual values first. Here are some examples. This can be placed in the azureaxsmod directory.
Note that the full path name to the script and variable files is included. You will need to adjust for your batch file. Create an access module load script, tdaxs4az load. The script will first drop any previous staging and error tables and then create our Vantage staging table.
Next, the script will perform a load to the staging table using an existing data file, dataset, in our Azure Blob Storage container data. Create a job variable jobvars file, jobvars load. This can be made much more complex in a production scenario, but it shows how additional processing can be performed.
Upload the files to Azure Blob Storage 1. Upload the previously created files to the container, files, in Azure Blob Storage.
BTEQ Input - Basic Teradata Query
Share your knowledge with other Developers in REVISIT CLASS.
Command Line Options - Basic Teradata Query
Connect Teradata Vantage to Azure Data Factory Using Custom Activity Feature