Semarchy Integrator : Working with FTP and SFTP

There is often a requirement to work with files using FTP and SFTP. This article highlights the Actions that can be used with these two protocols and how metadata connections are created and used.

The FTP, FTPS and SFTP Protocols

The following protocols exist to transfer files on a TCP/IP network.

• The File Transfer Protocol (FTP) is a standard network protocol used to transfer computer files from one host to another host.
• FTPS (also known as FTPES, FTP-SSL and FTP Secure) is an extension to the commonly used File Transfer Protocol (FTP) that adds support for the Transport Layer Security (TLS) and the Secure Sockets Layer (SSL) cryptographic protocols.
• The SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP) is a network protocol that enables secure file transfer and file management capabilities between networked hosts. SFTP uses SSH2 encryption to authenticate and establish secure channels between networked hosts. Although SFTP clients are functionally similar to FTP clients, they employ different protocols and therefore, you cannot use a standard FTP client to connect to an SFTP server.
• If you need to create a FTP server on EC2, here are stack overflow instruction to do it. 

Convergence for Data Integration Actions

Convergence for Data Integration provide the following actions to handle files these protocols:

• FTP and FTPS
  • Get Files with FTP
  • Send Files with FTP
  • Command with FTP

• SFTP

  • Get Files with SCP
  • Send Files with SCP
  • Execute SSH (to runs an SSH script on a remote host)

FTP/S and SFTP Metadata Connections

Metadata defined for an FTP/FTPS or an SFTP (SSH) connection is not designed to be used in the same way as a file or database metadata. It is used to contain the connection information to the server and its directories. You use such a metadata definition in FTP or SCP actions in processes, by dragging and dropping it on the action to create a metadata link. By doing so, you configure the action (host, port, user, password, etc.) with the parameters set in the metadata. You cannot reverse-engineer file structure through FTP/SFTP, as they are transfer protocols.

To reverse engineer files stored in a remote host, you should have a drive mapped to the remote location (folder) where those files are stored and use the regular File metadata reverse. If you are using both remote and local runtime engines, you would need a configuration for your remote server (the path from the remote runtime located in the remote location) and another one for your local runtime (the path from your local machine to the remote location).

Working with SFTP

To Use SFTP in Convergence for Data Integrator:

1. Create an SSH Metadata connection including the remote folder.
   
2. In the Process where the SFTP connection is required, drag and drop the SSH Metadata folder on to the SCP action to create a metadata link to this folder and make sure you select the Scp Use Sftp option in the properties
   

Working with FTP or FTPS

To use FTP or FTPS in Convergence for Data Integrator

1. Create an FTP Metadata Connection including the remote folder
   
2. In the Process where the FTP connection is used, drag and drop the folder on to the FTP action to create a metadata link to this folder.
   

Related articles