SFTPTransfer: Difference between revisions

From Commander4j
← Older edit
No edit summary
No edit summary
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
SFTP Transfer is a program designed to automate the movement of files to and from a SFTP server.  
SFTPTransfer is a background file transfer service that moves files between a local filesystem and a remote SFTP server. It is a standalone Java application that runs either as a headless background service or as a desktop application with a minimal GUI, and supports both uploading (PUT) and downloading (GET) in independent threads.


You can configure it :-
SFTPTransfer replaces the earlier sftpSend and sftpGet tools with a unified, configurable service.


Monitor a local folder for files and automatically move them to a remote server.
== Purpose ==
Monitor a remote SFTP folder and automatically move them to a local folder.


This program performs the same function as the [[SFTPSend]] and [[SFTPGet]] programs but makes things a lot easier to configure as it has a user interface as shown below.
SFTPTransfer automates recurring file transfers between Commander4j and external systems — for example, uploading despatch notifications to a customer portal or downloading order files from a supplier. It runs continuously, polling for new files at a configurable interval, and handles authentication, retry, backup, and archiving without operator involvement.


You can configure a test the program using the interactive display and then configure it to run as a background process.
[[File:SFTPTransfer.jpg|800px]]


== Startup Modes ==


The user interface consists of a number of tabs which contain both the settings and a log of activity.
SFTPTransfer supports two startup modes selected at launch time:


{| class="wikitable"
|-
! Mode !! Behaviour
|-
| '''Start desktop''' || Opens a small GUI window. Transfer threads start in a paused state; the operator must click Start in the interface to begin polling. Useful for testing configuration.
|-
| '''Start service''' || Headless — no GUI window. Transfer threads start immediately on launch. Suitable for production deployment as a background service or scheduled task.
|}


== Host Settings ==
On Windows, the native install package includes <code>sftpTransfer_Service.exe</code>, which manages the Windows service directly — see [[#Installing as a Windows Service|Installing as a Windows Service]] below. On Linux/macOS, use systemd, launchd, or nohup.
[[File:SFTP_Transfer_Common.png|border|800px]]


The first page used to define the SFTP host machine that you want to connect to. You can define it's IP Address and Port along with a known hosts file to validate it's authenticity and authentication details.
== Configuration Files ==


At the bottom of this page you can enable email notifications and protect the settings with a password.
All configuration is held in XML files under the <code>xml/config/</code> directory. Changes require a service restart unless hot-reload is triggered (see below).


== SFTP Put Settings ==
=== sftp_common.xml ===
[[File:SFTP_Transfer_Put_Properties.png|border|800px]]


The "Put Properties" allows you to define which local folder is to be monitored and where files should be sent to. To ensure no data is lost during transfer the system can also make backups of any files which it sends. To ensure that the backups do not accumulate forever you can also define a backup retention period. Files which have been backed up will be deleted when they are older that the backup retention period which is in "days".
Shared settings used by all transfer threads:


== SFTP Get Settings ==
{| class="wikitable"
[[File:SFTP_Transfer_Get_Properties.png|border|800px]]
|-
! Setting !! Description
|-
| Poll interval || How often (in seconds) each thread checks for new files. Default: 10 seconds.
|-
| Backup retention || Number of days to keep backed-up files before automatic deletion.
|-
| Log level || Logging verbosity passed to Log4j.
|}


== jsch Settings ==
=== sftp_put.xml ===
[[File:SFTP_Transfer_jsch_Properties.png|border|800px]]


== Email Settings ==
Configures one or more PUT (upload) profiles. Each profile defines:
[[File:SFTP_Transfer_Email_Properties.png|border|800px]]


== Put Log ==
{| class="wikitable"
[[File:SFTP_Transfer_Put_Log.png|border|800px]]
|-
! Setting !! Description
|-
| Enabled || Whether this profile's thread is active.
|-
| Local path || Local directory to watch for outbound files.
|-
| File mask || Wildcard pattern to match files (e.g. <code>*.xml</code>, <code>order_*.csv</code>). Uses Apache Commons IO WildcardFileFilter.
|-
| Remote host || SFTP server hostname or IP address.
|-
| Remote port || SFTP port (default: 22).
|-
| Remote path || Target directory on the remote server.
|-
| Username || SFTP login username.
|-
| Password || AES-encrypted password (see [[#Password Encryption|Password Encryption]]).
|-
| Private key path || Path to SSH private key file (alternative to password authentication).
|-
| Backup path || Local directory where successfully uploaded files are moved.
|-
| Error path || Local directory where files that failed to upload are moved.
|}


== Get Log ==
=== sftp_get.xml ===
[[File:SFTP_Transfer_Get_Log.png|border|800px]]


== System Log ==
Configures one or more GET (download) profiles. Each profile defines:
[[File:SFTP_Transfer_System_Log.png|border|800px]]
 
{| class="wikitable"
|-
! Setting !! Description
|-
| Enabled || Whether this profile's thread is active.
|-
| Remote host || SFTP server hostname or IP address.
|-
| Remote port || SFTP port (default: 22).
|-
| Remote path || Directory on the remote server to poll for incoming files.
|-
| File mask || Wildcard pattern matched against the remote file listing (server-side ls).
|-
| Local path || Local directory where downloaded files are saved.
|-
| Username || SFTP login username.
|-
| Password || AES-encrypted password.
|-
| Private key path || Path to SSH private key file.
|-
| Delete after download || Whether to delete the remote file after successful download.
|-
| Backup path || Local directory where a copy of each downloaded file is kept.
|}
 
=== jsch_properties.xml ===
 
Advanced SSH session parameters passed directly to the JSch library. Used to control host key checking, preferred algorithms, and connection timeouts. In most deployments this file can be left at its defaults.
 
=== email_properties.xml ===
 
SMTP settings for error notification emails. When a transfer failure occurs and email notification is enabled, SFTPTransfer sends an alert to the configured address. Fields include SMTP host, port, sender address, recipient address, and AES-encrypted SMTP password.
 
== Installing as a Windows Service ==
 
The native install package includes <code>sftpTransfer_Service.exe</code>, a service-mode launcher built with install4j. This executable manages the Windows service directly — no additional wrapper is required.
 
All service commands must be run from an elevated command prompt (Run as Administrator).
 
The default Windows service name is '''sftpTransfer_Service''' (the launcher name defined in the installer). You can specify a different name as an optional second parameter — useful if you need to run multiple instances on the same machine. If a custom name is used at install time, that same name must be passed to every subsequent command.
 
{| class="wikitable"
|-
! Command !! Effect
|-
| <code>sftpTransfer_Service.exe /install</code> || Register as '''sftpTransfer_Service''' with automatic startup on boot
|-
| <code>sftpTransfer_Service.exe /install "My SFTP Service"</code> || Register with a custom service name
|-
| <code>sftpTransfer_Service.exe /install-demand</code> || Register with manual startup only
|-
| <code>sftpTransfer_Service.exe /start</code> || Start the service
|-
| <code>sftpTransfer_Service.exe /stop</code> || Stop the service (waits for graceful shutdown)
|-
| <code>sftpTransfer_Service.exe /restart</code> || Restart the service
|-
| <code>sftpTransfer_Service.exe /status</code> || Query status (exit code: 0 = running, 3 = stopped)
|-
| <code>sftpTransfer_Service.exe /uninstall</code> || Remove the service registration
|}
 
If a custom service name was used at install time, append it to all subsequent commands:
 
sftpTransfer_Service.exe /start "My SFTP Service"
sftpTransfer_Service.exe /stop "My SFTP Service"
sftpTransfer_Service.exe /uninstall "My SFTP Service"
 
Once installed, the service also appears in the Windows Services management console (<code>services.msc</code>).
 
=== Testing Before Installing as a Service ===
 
The native install package also provides two alternatives for testing:
 
{| class="wikitable"
|-
! Launcher !! Mode !! Use case
|-
| <code>sftpTransfer_Console.exe</code> || Console (headless) || Runs the service logic in a visible console window — useful for watching log output during initial configuration
|-
| <code>sftpTransfer.exe</code> || Desktop GUI || Opens the GUI window in Start desktop mode — transfer threads start paused, allowing configuration to be verified before enabling transfers
|}
 
== Authentication ==
 
SFTPTransfer supports two authentication methods, configured per profile:
 
{| class="wikitable"
|-
! Method !! How to configure
|-
| Password || Set the <code>Password</code> field in the profile to an AES-encrypted value. Leave the private key path empty.
|-
| SSH private key || Set the <code>Private key path</code> field to the path of a PEM-format private key file. The password field is ignored.
|}
 
Both methods use the JSch 2.x SSH library with Bouncy Castle as the cryptographic provider.
 
=== Password Encryption ===
 
Passwords stored in configuration files are AES-encrypted. Plain-text passwords are never stored on disk. To encrypt a password, use the password utility included with Commander4j (the same utility used for database passwords in Commander4j's own configuration).
 
== File Transfer Behaviour ==
 
=== PUT (Upload) ===
 
# SFTPTransfer polls the local input directory every 10 seconds (configurable).
# Files matching the file mask are selected for upload.
# Each file is uploaded to the remote path using a temporary name with a <code>.tmp</code> extension.
# On successful transfer, the remote file is renamed to its final name (atomic rename — the remote system never sees a partial file).
# The local source file is moved to the backup directory.
# If the transfer fails, the file is moved to the error directory and an error is logged.
 
=== GET (Download) ===
 
# SFTPTransfer lists the remote directory every 10 seconds (configurable).
# Files matching the file mask are selected for download.
# Each file is downloaded to the local path using a <code>.tmp</code> extension.
# On successful download, the local file is renamed to its final name.
# If configured, the remote file is deleted after download.
# A backup copy is optionally saved to the local backup directory.
 
== Thread Architecture ==
 
When running, SFTPTransfer maintains the following threads:
 
{| class="wikitable"
|-
! Thread !! Purpose
|-
| TransferPUT || One thread per enabled PUT profile, each managing its own SFTP connection.
|-
| TransferGET || One thread per enabled GET profile, each managing its own SFTP connection.
|-
| EmailThread || Sends error notification emails asynchronously.
|-
| ArchiveThread || Periodically purges backup files older than the configured retention period.
|}
 
== Hot Configuration Reload ==
 
SFTPTransfer supports a configuration reload mode (<code>Mode_CONFIG_UPDATE</code>). When this mode is triggered, running threads re-read their configuration files without requiring a full service restart. This allows profile changes — such as updating a remote path or rotating credentials — to be applied with minimal disruption.
 
== Logging ==
 
All activity is written to the Log4j log file. The log records:
 
* Service startup and configuration loaded
* Each thread starting and its profile settings
* Every file detected, transfer attempted, and outcome
* Every remote connection attempt and disconnect
* File move operations (to backup or error directories)
* Errors and exceptions with full context
 
Log rotation and retention are configured in the Log4j configuration file included in the distribution.
 
== Dependencies ==
 
{| class="wikitable"
|-
! Library !! Version !! Purpose
|-
| JSch || 2.27.9 || SSH/SFTP client
|-
| Bouncy Castle || current || Cryptographic provider for JSch
|-
| Apache Commons IO || current || WildcardFileFilter for local file matching
|-
| Log4j || 2.25.3 || Logging
|}
 
See also: [[LabelServer4j]], [[Middleware4j]], [[Production Lines & Labellers]]
 
[[Category:Commander4j]]

Latest revision as of 19:53, 24 April 2026

SFTPTransfer is a background file transfer service that moves files between a local filesystem and a remote SFTP server. It is a standalone Java application that runs either as a headless background service or as a desktop application with a minimal GUI, and supports both uploading (PUT) and downloading (GET) in independent threads.

SFTPTransfer replaces the earlier sftpSend and sftpGet tools with a unified, configurable service.

Purpose

SFTPTransfer automates recurring file transfers between Commander4j and external systems — for example, uploading despatch notifications to a customer portal or downloading order files from a supplier. It runs continuously, polling for new files at a configurable interval, and handles authentication, retry, backup, and archiving without operator involvement.

Startup Modes

SFTPTransfer supports two startup modes selected at launch time:

Mode Behaviour
Start desktop Opens a small GUI window. Transfer threads start in a paused state; the operator must click Start in the interface to begin polling. Useful for testing configuration.
Start service Headless — no GUI window. Transfer threads start immediately on launch. Suitable for production deployment as a background service or scheduled task.

On Windows, the native install package includes sftpTransfer_Service.exe, which manages the Windows service directly — see Installing as a Windows Service below. On Linux/macOS, use systemd, launchd, or nohup.

Configuration Files

All configuration is held in XML files under the xml/config/ directory. Changes require a service restart unless hot-reload is triggered (see below).

sftp_common.xml

Shared settings used by all transfer threads:

Setting Description
Poll interval How often (in seconds) each thread checks for new files. Default: 10 seconds.
Backup retention Number of days to keep backed-up files before automatic deletion.
Log level Logging verbosity passed to Log4j.

sftp_put.xml

Configures one or more PUT (upload) profiles. Each profile defines:

Setting Description
Enabled Whether this profile's thread is active.
Local path Local directory to watch for outbound files.
File mask Wildcard pattern to match files (e.g. *.xml, order_*.csv). Uses Apache Commons IO WildcardFileFilter.
Remote host SFTP server hostname or IP address.
Remote port SFTP port (default: 22).
Remote path Target directory on the remote server.
Username SFTP login username.
Password AES-encrypted password (see Password Encryption).
Private key path Path to SSH private key file (alternative to password authentication).
Backup path Local directory where successfully uploaded files are moved.
Error path Local directory where files that failed to upload are moved.

sftp_get.xml

Configures one or more GET (download) profiles. Each profile defines:

Setting Description
Enabled Whether this profile's thread is active.
Remote host SFTP server hostname or IP address.
Remote port SFTP port (default: 22).
Remote path Directory on the remote server to poll for incoming files.
File mask Wildcard pattern matched against the remote file listing (server-side ls).
Local path Local directory where downloaded files are saved.
Username SFTP login username.
Password AES-encrypted password.
Private key path Path to SSH private key file.
Delete after download Whether to delete the remote file after successful download.
Backup path Local directory where a copy of each downloaded file is kept.

jsch_properties.xml

Advanced SSH session parameters passed directly to the JSch library. Used to control host key checking, preferred algorithms, and connection timeouts. In most deployments this file can be left at its defaults.

email_properties.xml

SMTP settings for error notification emails. When a transfer failure occurs and email notification is enabled, SFTPTransfer sends an alert to the configured address. Fields include SMTP host, port, sender address, recipient address, and AES-encrypted SMTP password.

Installing as a Windows Service

The native install package includes sftpTransfer_Service.exe, a service-mode launcher built with install4j. This executable manages the Windows service directly — no additional wrapper is required.

All service commands must be run from an elevated command prompt (Run as Administrator).

The default Windows service name is sftpTransfer_Service (the launcher name defined in the installer). You can specify a different name as an optional second parameter — useful if you need to run multiple instances on the same machine. If a custom name is used at install time, that same name must be passed to every subsequent command.

Command Effect
sftpTransfer_Service.exe /install Register as sftpTransfer_Service with automatic startup on boot
sftpTransfer_Service.exe /install "My SFTP Service" Register with a custom service name
sftpTransfer_Service.exe /install-demand Register with manual startup only
sftpTransfer_Service.exe /start Start the service
sftpTransfer_Service.exe /stop Stop the service (waits for graceful shutdown)
sftpTransfer_Service.exe /restart Restart the service
sftpTransfer_Service.exe /status Query status (exit code: 0 = running, 3 = stopped)
sftpTransfer_Service.exe /uninstall Remove the service registration

If a custom service name was used at install time, append it to all subsequent commands: 

sftpTransfer_Service.exe /start "My SFTP Service"
sftpTransfer_Service.exe /stop "My SFTP Service"
sftpTransfer_Service.exe /uninstall "My SFTP Service"

Once installed, the service also appears in the Windows Services management console (services.msc).

Testing Before Installing as a Service

The native install package also provides two alternatives for testing:

Launcher Mode Use case
sftpTransfer_Console.exe Console (headless) Runs the service logic in a visible console window — useful for watching log output during initial configuration
sftpTransfer.exe Desktop GUI Opens the GUI window in Start desktop mode — transfer threads start paused, allowing configuration to be verified before enabling transfers

Authentication

SFTPTransfer supports two authentication methods, configured per profile:

Method How to configure
Password Set the Password field in the profile to an AES-encrypted value. Leave the private key path empty.
SSH private key Set the Private key path field to the path of a PEM-format private key file. The password field is ignored.

Both methods use the JSch 2.x SSH library with Bouncy Castle as the cryptographic provider.

Password Encryption

Passwords stored in configuration files are AES-encrypted. Plain-text passwords are never stored on disk. To encrypt a password, use the password utility included with Commander4j (the same utility used for database passwords in Commander4j's own configuration).

File Transfer Behaviour

PUT (Upload)

  1. SFTPTransfer polls the local input directory every 10 seconds (configurable).
  2. Files matching the file mask are selected for upload.
  3. Each file is uploaded to the remote path using a temporary name with a .tmp extension.
  4. On successful transfer, the remote file is renamed to its final name (atomic rename — the remote system never sees a partial file).
  5. The local source file is moved to the backup directory.
  6. If the transfer fails, the file is moved to the error directory and an error is logged.

GET (Download)

  1. SFTPTransfer lists the remote directory every 10 seconds (configurable).
  2. Files matching the file mask are selected for download.
  3. Each file is downloaded to the local path using a .tmp extension.
  4. On successful download, the local file is renamed to its final name.
  5. If configured, the remote file is deleted after download.
  6. A backup copy is optionally saved to the local backup directory.

Thread Architecture

When running, SFTPTransfer maintains the following threads:

Thread Purpose
TransferPUT One thread per enabled PUT profile, each managing its own SFTP connection.
TransferGET One thread per enabled GET profile, each managing its own SFTP connection.
EmailThread Sends error notification emails asynchronously.
ArchiveThread Periodically purges backup files older than the configured retention period.

Hot Configuration Reload

SFTPTransfer supports a configuration reload mode (Mode_CONFIG_UPDATE). When this mode is triggered, running threads re-read their configuration files without requiring a full service restart. This allows profile changes — such as updating a remote path or rotating credentials — to be applied with minimal disruption.

Logging

All activity is written to the Log4j log file. The log records:

  • Service startup and configuration loaded
  • Each thread starting and its profile settings
  • Every file detected, transfer attempted, and outcome
  • Every remote connection attempt and disconnect
  • File move operations (to backup or error directories)
  • Errors and exceptions with full context

Log rotation and retention are configured in the Log4j configuration file included in the distribution.

Dependencies

Library Version Purpose
JSch 2.27.9 SSH/SFTP client
Bouncy Castle current Cryptographic provider for JSch
Apache Commons IO current WildcardFileFilter for local file matching
Log4j 2.25.3 Logging

See also: LabelServer4j, Middleware4j, Production Lines & Labellers

Retrieved from ‘https://wiki.commander4j.com/index.php?title=SFTPTransfer&oldid=1829’