Jump to content

SFTPTransfer: Difference between revisions

From Commander4j
No edit summary
Updated by push_wiki.py
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
SFTPTransfer is a background file transfer service that moves files between a local filesystem and a remote SFTP server. It is a standalone Java application that runs either as a headless background service or as a desktop application with a minimal GUI, and supports both uploading (PUT) and downloading (GET) in independent threads.
'''SFTPTransfer''' is a general purpose file transfer utility that moves or synchronises files between a local filesystem and a remote SFTP server. It is a standalone Java application with no dependency on any other Commander4j component: it can be used on its own wherever files need to be exchanged with an SFTP server on a schedule, and it runs on any platform with a Java runtime. It runs either as a headless background service or as a desktop application with a settings window and live log, and it uploads (Put) and downloads (Get) in independent threads.


SFTPTransfer replaces the earlier sftpSend and sftpGet tools with a unified, configurable service.
SFTPTransfer replaces the earlier [[SFTPSend]] and [[SFTPGet]] tools with a single configurable service. Sites running [[Commander4j]] or [[Middleware4j]] typically use it to exchange files with ERP systems, customer portals, suppliers and print-and-apply labellers, but nothing in the program assumes those uses.
 
This page describes version 6.19.
 
[[File:SFTPTransfer.jpg|800px]]


== Purpose ==
== Purpose ==


SFTPTransfer automates recurring file transfers between Commander4j and external systems — for example, uploading despatch notifications to a customer portal or downloading order files from a supplier. It runs continuously, polling for new files at a configurable interval, and handles authentication, retry, backup, and archiving without operator involvement.
SFTPTransfer polls a local folder (Put) and a remote folder (Get) at a configurable interval and transfers any file that matches a file mask. Each direction works in one of two modes:


[[File:SFTPTransfer.jpg|800px]]
{| class="wikitable"
|-
! Mode !! Behaviour
|-
| '''Move''' || Every matching file is copied to the other side and the source copy is deleted once the copy has been renamed into place. This is the default (Sync Mode unticked). Put can keep a local backup copy of each file it sends.
|-
| '''Sync''' || One-way mirror. New or changed files are copied, the source is left in place and the modified time is carried across so the file is not copied again. Optionally, files and folders that have disappeared from the source are deleted from the destination.
|}
 
It handles authentication, retry, backup, archiving, logging and email notification without operator involvement.


== Startup Modes ==
== Startup Modes ==


SFTPTransfer supports two startup modes selected at launch time:
SFTPTransfer supports two startup modes selected by the first command line argument:
 
{| class="wikitable"
|-
! Mode !! Argument !! Behaviour
|-
| '''Desktop''' || <code>desktop</code> || Opens the settings window. Transfer threads start paused; press Run in the toolbar to begin polling. Used to configure the program and to watch transfers.
|-
| '''Service''' || <code>service</code> || Headless. Transfer threads start immediately and run until the process is stopped. Used for production deployment as a background service.
|}
 
The program reads its configuration from <code>./xml/config</code> relative to the working directory, so it must be started from its installation folder. To run it by hand:
 
java -Dlog4j.configurationFile=./xml/config/log4j2.xml -jar sftpTransfer.jar desktop
java -Dlog4j.configurationFile=./xml/config/log4j2.xml -jar sftpTransfer.jar service
 
The native install packages provide launchers for each mode:
 
{| class="wikitable"
|-
! Launcher !! Mode !! Use
|-
| <code>sftpTransfer</code> || Desktop || Settings window and live log.
|-
| <code>sftpTransfer_Console</code> || Service, in a console window || Runs the service logic with the log visible on screen. Useful during initial configuration.
|-
| <code>sftpTransfer_Service</code> || Service || On Windows this executable registers and controls the Windows service, see [[#Installing as a Windows Service|Installing as a Windows Service]]. On Linux and macOS use systemd, launchd or nohup with the <code>service</code> argument.
|}
 
A desktop window and a service can run on the same machine at the same time, see [[#Running the Desktop Beside a Service|Running the Desktop Beside a Service]].
 
== Installation Folder ==
 
{| class="wikitable"
|-
! Folder or file !! Contents
|-
| <code>xml/config/</code> || The configuration files described below, plus the runtime files <code>reload.request</code> and <code>service.lock</code>.
|-
| <code>xml/config/examples/</code> || Neutral example copies of every configuration file.
|-
| <code>ssh/</code> || <code>known_hosts</code> and, if key authentication is used, the private and public key files.
|-
| <code>safety/</code> || The template <code>local.safe</code> and <code>remote.safe</code> files copied by the Create buttons, see [[#Safe Files|Safe Files]].
|-
| <code>logs/</code> || <code>sftpTransfer.log</code> and its rolled-over archives.
|-
| <code>send/</code>, <code>receive/</code>, <code>backups/</code> || Default local, download and backup folders used by the example configuration. Any folder can be configured instead.
|}
 
== Desktop Window ==
 
The window has the settings tabs on the left and a combined log on the right, separated by a draggable divider.
 
=== Tabs ===
 
{| class="wikitable"
|-
! Tab !! Settings
|-
| '''SFTP''' || Title, SFTP host and port, known hosts handling, authentication type, SFTP username and password, private and public key files, email, daily statistics and web log viewer switches, settings password. Saved in <code>sftp_common.xml</code>.
|-
| '''Put Properties''' || Upload settings. Saved in <code>sftp_put.xml</code>.
|-
| '''Get Properties''' || Download settings. Saved in <code>sftp_get.xml</code>.
|-
| '''jsch Properties''' || Advanced SSH session parameters passed to the JSch library. Saved in <code>jsch_properties.xml</code>.
|-
| '''Email Properties''' || SMTP properties and the three distribution lists, plus a Send Test Email button. Saved in <code>email_properties.xml</code> and <code>email_distribution_properties.xml</code>.
|}
 
=== Toolbar ===
 
{| class="wikitable"
|-
! Button !! Action
|-
| '''Run / Pause''' || Starts or pauses the transfer threads of this window. A pass that is under way stops at the next file boundary: the file in flight is finished, the remaining files wait for the next poll, and the pass summary and Transfer email record that the pass was paused. Pausing closes the SFTP sessions. Disabled while a service is running on the same machine.
|-
| '''Apply''' || Saves the settings on screen to <code>xml/config</code>, hands them to the running threads and writes a reload request for any service running on this machine. Enabled only when the form differs from what is on disk.
|-
| '''Reload''' || Re-reads the saved files into the form, discarding edits that have not been applied.
|-
| '''Help''' || Opens this page in the default browser.
|-
| '''About''' and '''Licences''' || Version and third party licence information.
|}
 
Settings are taken up by each thread between passes: a transfer pass that is in progress finishes with the old settings, including any sync deletions, and the status line shows "Waiting for the current transfer to complete" until both threads have changed over. Only Pause and a service stop interrupt a pass; Apply never does. Every Apply drops the SFTP sessions and reconnects with the new settings on the next pass.
 
The Local Folder, Remote Folder and Backup Folder fields each have a browse button. The remote browse connects with the SFTP settings as typed on the SFTP tab, so the paths can be chosen before anything has been applied or run.
 
=== Log Panel ===
 
The log panel shows the Put, Get and System logs merged in time order, each line prefixed <code>[PUT]</code>, <code>[GET]</code> or <code>[SYS]</code>. Long lines wrap. The strip above the log holds:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Mode !! Behaviour
! Control !! Action
|-
| '''System / Put / Get''' || Hide or show each log. Hidden lines are kept and reappear when the box is ticked again. All three are ticked at every start.
|-
| '''Clear''' || Empties the log window. The log file and the web viewer are not affected.
|-
|-
| '''Start desktop''' || Opens a small GUI window. Transfer threads start in a paused state; the operator must click Start in the interface to begin polling. Useful for testing configuration.
| '''Save''' || Writes the lines currently shown to a text file (default <code>logs/sftpTransfer_window_<date>.txt</code>).
|-
|-
| '''Start service''' || Headless — no GUI window. Transfer threads start immediately on launch. Suitable for production deployment as a background service or scheduled task.
| '''Log Level''' || Errors only, Warnings and errors, Normal or Debug. Applies to new lines. Normal hides the polling of the remote folders; Debug shows every <code>ls</code>. The choice is remembered in <code>sftp_common.xml</code>. A running service keeps its level until the next Apply.
|}
|}


On Windows, the native install package includes <code>sftpTransfer_Service.exe</code>, which manages the Windows service directly — see [[#Installing as a Windows Service|Installing as a Windows Service]] below. On Linux/macOS, use systemd, launchd, or nohup.
Two progress bars beneath the strip, Put (orange) and Get (blue), show the file being transferred, the percentage and the bytes so far. A finished file stays on show until the next one starts.


== Configuration Files ==
== Configuration Files ==


All configuration is held in XML files under the <code>xml/config/</code> directory. Changes require a service restart unless hot-reload is triggered (see below).
All configuration is held in XML files under <code>xml/config/</code>. Every value has an <code>encrypted</code> attribute; passwords are stored encrypted and are set from the desktop window, never typed into the file. Settings added in later versions are optional: a file written by an earlier version is read with the defaults shown below.


=== sftp_common.xml ===
=== sftp_common.xml ===


Shared settings used by all transfer threads:
The connection and the program-wide switches, shared by Put and Get:


{| class="wikitable"
{| class="wikitable"
Line 36: Line 147:
! Setting !! Description
! Setting !! Description
|-
|-
| Poll interval || How often (in seconds) each thread checks for new files. Default: 10 seconds.
| Title || Free text shown in the window title, the web page and email subjects.
|-
| Host Address, Port || SFTP server and port (default 22). There is one server: Put and Get both use it.
|-
|-
| Backup retention || Number of days to keep backed-up files before automatic deletion.
| Known Hosts File, Add to Known Hosts File || Path to the <code>known_hosts</code> file (default <code>./ssh/known_hosts</code>) and whether an unknown host key is added automatically.
|-
|-
| Log level || Logging verbosity passed to Log4j.
| Authentication Type || <code>user password</code> or <code>user public key</code>.
|-
| SFTP Username, SFTP Password || SFTP login. The password is set with the Assign SFTP user password button and stored encrypted.
|-
| Private Key File, Public Key File, Key Comment, Private Key Password || Key pair for public key authentication (default <code>./ssh/sftpTransfer</code> and <code>.pub</code>). The Create button generates a new pair. The key password is set with its Assign button and stored encrypted.
|-
| Email Notifications || Master switch for all email, see [[#Email Notifications|Email Notifications]]. Default off.
|-
| Daily Statistics Email, at || Sends the day's transfer counts to the Monitor list once a day at the given time (HH:mm, '''UTC''', not this machine's clock, so servers in different time zones report on the same day boundary). Default off, 06:00.
|-
| Web Log Viewer, Port || Serves the logs as a read-only web page, see [[#Web Log Viewer|Web Log Viewer]]. Default off, port 8080.
|-
| SettingsPassword || Optional. When set, the desktop window asks for it at start-up before the settings can be seen or changed. Set and cleared with its own buttons.
|-
| Log Level || The desktop log level, written whenever the combo is changed.
|}
|}


=== sftp_put.xml ===
=== sftp_put.xml ===


Configures one or more PUT (upload) profiles. Each profile defines:
The upload (Put) settings:


{| class="wikitable"
{| class="wikitable"
Line 51: Line 178:
! Setting !! Description
! Setting !! Description
|-
|-
| Enabled || Whether this profile's thread is active.
| Enabled || Whether the Put thread transfers anything. A disabled side logs a warning at start-up and does nothing.
|-
|-
| Local path || Local directory to watch for outbound files.
| Title || Free text, shown in the Put log and emails.
|-
|-
| File mask || Wildcard pattern to match files (e.g. <code>*.xml</code>, <code>order_*.csv</code>). Uses Apache Commons IO WildcardFileFilter.
| Local Folder || Folder watched for outbound files. Must contain <code>local.safe</code>.
|-
|-
| Remote host || SFTP server hostname or IP address.
| Local File Mask || One or more wildcard masks separated by <code>;</code>, for example <code>*.xml;*.csv</code>. See [[#File Masks|File Masks]].
|-
|-
| Remote port || SFTP port (default: 22).
| Backup Folder, Backup Retention || Move mode only. Each sent file is copied here (sub folder structure preserved) before the local copy is deleted, and copies older than the retention period in days are purged. Greyed out in Sync Mode. The purge only runs when a marker file named <code>backup.folder</code> exists in the backup folder.
|-
|-
| Remote path || Target directory on the remote server.
| Poll Frequency || Seconds between passes, 1 to 86400 (one day). A value outside that range is refused by Apply with a message naming the field. Default 10.
|-
|-
| Username || SFTP login username.
| Remote Folder || Destination folder on the server. Must contain <code>remote.safe</code>.
|-
|-
| Password || AES-encrypted password (see [[#Password Encryption|Password Encryption]]).
| Temporary File Extension || Files are uploaded under this extension (default <code>.tmp</code>) and renamed on completion, so the remote side never sees a partial file. Files carrying the extension are never selected for transfer.
|-
|-
| Private key path || Path to SSH private key file (alternative to password authentication).
| Include Sub Folders || Also send files from folders beneath the local folder, recreating the structure on the server. Default off.
|-
|-
| Backup path || Local directory where successfully uploaded files are moved.
| Sync Mode || Copy new or changed files and leave the local files in place. Unticked means move mode. Default off.
|-
|-
| Error path || Local directory where files that failed to upload are moved.
| Overwrite Newer || Sync only. Also replace a newer remote file with an older local one. Default on.
|-
| Sync Deletions || Sync only. Also delete remote files which are no longer in the local folder. Default off.
|-
| Sync Folder Deletions || Sync only, needs Include Sub Folders and Sync Deletions. Also remove remote folders which are no longer in the local folder. Default off.
|-
| Stability Check, Seconds, Retries || Only transfer a file once it has stopped changing for the given number of seconds. A file that is still changing after the wait is checked again, up to Retries times, each after another full wait; only then is it skipped until the next poll. Retries 0 restores the single check. Default off, 5 seconds, 5 retries (an installation upgraded from an earlier version gets 5). With the defaults one file that keeps growing can hold a pass for up to six times the stability seconds.
|}
|}


=== sftp_get.xml ===
=== sftp_get.xml ===


Configures one or more GET (download) profiles. Each profile defines:
The download (Get) settings have the same options seen from the other side:


{| class="wikitable"
{| class="wikitable"
Line 82: Line 215:
! Setting !! Description
! Setting !! Description
|-
|-
| Enabled || Whether this profile's thread is active.
| Enabled || Whether the Get thread transfers anything.
|-
| Title || Free text, shown in the Get log and emails.
|-
| Remote Folder || Folder on the server polled for incoming files. Must contain <code>remote.safe</code>.
|-
|-
| Remote host || SFTP server hostname or IP address.
| Remote File Mask || One or more wildcard masks separated by <code>;</code>, matched against the remote listing.
|-
|-
| Remote port || SFTP port (default: 22).
| Local Folder || Folder where downloaded files are saved. Must contain <code>local.safe</code>.
|-
|-
| Remote path || Directory on the remote server to poll for incoming files.
| Poll Frequency || Seconds between passes, 1 to 86400 (one day). A value outside that range is refused by Apply with a message naming the field. Default 10.
|-
|-
| File mask || Wildcard pattern matched against the remote file listing (server-side ls).
| Temporary File Extension || Files are downloaded under this extension and renamed on completion.
|-
|-
| Local path || Local directory where downloaded files are saved.
| Include Sub Folders || Also fetch files from folders beneath the remote folder.
|-
|-
| Username || SFTP login username.
| Sync Mode || Copy new or changed files and leave the remote files in place. Unticked means move mode (the remote file is deleted after a completed download).
|-
|-
| Password || AES-encrypted password.
| Overwrite Newer || Sync only. Also replace a newer local file with an older remote one. Default on.
|-
|-
| Private key path || Path to SSH private key file.
| Sync Deletions || Sync only. Also delete local files which are no longer in the remote folder.
|-
|-
| Delete after download || Whether to delete the remote file after successful download.
| Sync Folder Deletions || Sync only. Also remove local folders which are no longer in the remote folder.
|-
|-
| Backup path || Local directory where a copy of each downloaded file is kept.
| Stability Check, Seconds, Retries || Only fetch a file once the remote listing shows it unchanged for the given number of seconds. A file that is still changing is listed again, up to Retries times, each after another full wait, before it is skipped until the next poll. Retries 0 restores the single check. Default off, 5 seconds, 5 retries.
|}
|}


=== jsch_properties.xml ===
=== jsch_properties.xml ===


Advanced SSH session parameters passed directly to the JSch library. Used to control host key checking, preferred algorithms, and connection timeouts. In most deployments this file can be left at its defaults.
Session parameters passed directly to the JSch library: strict host key checking, preferred authentication methods, key exchange, cipher, MAC and host key algorithm lists, compression and proxy settings. Each row can be enabled or disabled and encrypted individually. The shipped defaults suit most servers; change them only when a server insists on a particular algorithm.


=== email_properties.xml ===
=== email_properties.xml ===


SMTP settings for error notification emails. When a transfer failure occurs and email notification is enabled, SFTPTransfer sends an alert to the configured address. Fields include SMTP host, port, sender address, recipient address, and AES-encrypted SMTP password.
Jakarta Mail SMTP properties. Six rows are always present and cannot be removed: <code>mail.smtp.host</code>, <code>mail.smtp.port</code>, <code>mail.smtp.from</code>, <code>mail.smtp.auth</code>, <code>mail.smtp.user</code> and <code>mail.smtp.password</code>; a file that lacks any of them gains the row, disabled and empty, the next time the desktop is opened. The '''Add Property''' button beside Send Test Email offers the usual TLS, SSL, socket factory, timeout, charset and debug properties with suggested values, or any other name typed in; '''Remove Property''' deletes the selected row and refuses the six required ones. The password row is stored encrypted. Rows with Enable unticked are left out of the session. A server that answers <code>530 ... anonymous mail</code> or a similar authentication error usually needs <code>mail.smtp.auth</code> enabled and set to <code>true</code>.
 
=== email_distribution_properties.xml ===
 
The three distribution lists. Each has an address list (comma separated), an Enable flag and a Frequency in minutes, see [[#Email Notifications|Email Notifications]].
 
=== log4j2.xml ===
 
Log file location, rollover and retention, see [[#Logging|Logging]].
 
== Safe Files ==
 
A transfer pass does nothing unless both of these files exist:
 
{| class="wikitable"
|-
! File !! Where
|-
| <code>local.safe</code> || In the root of the Local Folder (Put and Get each have their own).
|-
| <code>remote.safe</code> || In the root of the Remote Folder on the server.
|}
 
The files prove that the configured folders are the intended ones and that any network share or mount behind them is actually present. Without them a mistyped path, an unmounted share or an empty folder could otherwise look like "nothing to send" or, with Sync Deletions on, empty the other side. Only the name matters, not the content.
 
* The Put and Get tabs have Create (tick) and Remove (cross) buttons beside the Local Folder and Remote Folder fields. Create copies the template from the <code>safety</code> folder into the folder typed in the field, after showing the full path and asking for confirmation. The remote buttons open their own SFTP session.
* A missing safe file is logged once as a warning (<code>Nothing sent, safe file not found : ...</code>) and again after Apply or a restart; transfers resume at the next poll once the file appears.
* The safe files are never transferred, deleted or counted, whatever the file mask.
* The remote check lists the remote folder, so the SFTP account must be permitted to list it.
* Create the local file only while any share it lives on is mounted, otherwise the empty mount point is what gets approved.
 
When upgrading from a version before 6.10, nothing transfers until the safe files have been created. Where the remote server is owned by another party they will need to place <code>remote.safe</code> for you.
 
== File Masks ==
 
The Local File Mask (Put) and Remote File Mask (Get) accept the wildcards <code>*</code> and <code>?</code>. Several masks can be given separated by <code>;</code>, for example <code>*.xml;*.csv</code>; spaces around a mask are ignored.
 
* Put matching is case-insensitive; Get matching follows the server listing and is case-sensitive.
* A blank mask matches nothing on both sides.
* Files carrying the Temporary File Extension are always skipped, even with a mask of <code>*</code>.
* A folder whose name matches the mask is skipped, never transferred or deleted.
* The mask is also the scope of Sync Deletions: only files matching it are deleted from the destination.
 
== Transfer Behaviour ==
 
=== Move Mode ===
 
Move mode is in force when Sync Mode is unticked.
 
# Each poll collects the files matching the mask (and, with Include Sub Folders, those in folders beneath).
# With Stability Check on, the thread waits until the collection is the configured number of seconds old, then re-checks each file's size and modified time. A file that has changed is waited for again, up to the configured number of Retries (logged as <code>Still changing ... retry n of m</code>); a file that is still changing after the last retry is skipped with a warning and picked up on the next poll.
# Put: the file is copied to the backup folder if backup is on, uploaded under the temporary extension, renamed to its final name, and only then deleted locally. Get: the file is downloaded under the temporary extension, renamed, and only then deleted from the server. A partial transfer is removed and the source is left in place.
# Sub folders are created on the destination as needed. The configured folders themselves are never created or removed, and emptied source sub folders are left in place.
# A file that keeps failing is retried and logged on every poll. There is no error folder.
 
=== Sync Mode ===
 
Sync Mode is a one-way mirror from source to destination; the source is never deleted. For each file the destination copy is compared by size and modified time:
 
{| class="wikitable"
|-
! Destination copy !! Action
|-
| Missing, older, or same time with a different size || Transferred. The modified time is carried across.
|-
| Identical (same size and time) || Skipped silently.
|-
| Newer than the source || Transferred only when Overwrite Newer is ticked. Otherwise kept, logged at Debug level only.
|}
 
With Include Sub Folders on, empty folders are created on the destination too.
 
'''Sync Deletions''': at the end of each folder, a destination file is deleted when it matches the mask, does not carry the temporary extension and no file of that name exists in the matching source folder. Folders are never removed by this option, and folders that exist only at the destination are left alone with their contents. Nothing is deleted when a folder cannot be listed or when the pass is interrupted. An empty source folder (with its safe file present) empties the destination of mask-matching files.
 
'''Sync Folder Deletions''' (needs Include Sub Folders and Sync Deletions): a folder that exists only at the destination has its mask-matching files deleted and is removed once it is empty. A folder still holding a file the mask does not cover is kept with that file. A folder is only judged to be destination-only when nothing of that name exists at the source, including a file or a link.
 
Each folder is processed in full (transfers, then deletions) before the next one is started, and a log line <code>Folder <path></code> is written before the first action in a folder.
 
Points to note:
 
* Do not tick Sync Deletions on both Put and Get against the same pair of folders.
* Sync Folder Deletions treats every folder beneath the destination as in scope. A "processed" folder created on the server by another system, or an archive folder beneath the receive folder, will lose its mask-matching files.
* On a Mac, Finder's <code>.DS_Store</code> file does not match a mask such as <code>*.xml</code>, so a browsed folder is emptied but not removed.
* Backup is for move mode: in Sync Mode the source file is still there, so the backup row is greyed out and no backup copy is taken.
 
=== Pass Summary ===
 
Every pass that transferred, failed or skipped at least one file ends with <code>Transfer finished - n sent, n failed, n skipped in n s</code>. A pass that does not run to its end says instead <code>Transfer interrupted by shutdown</code>, <code>Transfer paused from the window</code> or <code>Transfer abandoned, connection lost</code>, followed by the same counts, and the Transfer email for that pass carries an <code>Outcome</code> row saying so and listing the files that did get through; the rest are sent or fetched on the next poll. A pass with nothing to do writes nothing at the Normal log level, so an idle service produces a quiet log.
 
Each thread disconnects from the server at the end of every pass and reconnects at the start of the next, so a long poll interval never leaves an idle session for a firewall or the server to drop. The log therefore shows Connecting, Connected and Disconnected on every pass, including an idle one.
 
== Applying Settings to a Running Service ==
 
A service reads <code>xml/config</code> at start-up and whenever the file <code>xml/config/reload.request</code> appears. The desktop's Apply button writes the settings and then writes that file; a service on the same machine collects it within a second, re-reads the six configuration files, hands the settings to its threads (between passes) and deletes the file. The desktop status line reports whether a service collected the request.
 
An empty <code>reload.request</code> created by any other means works too, so settings prepared on another machine can be copied into place and activated without restarting the service. If any configuration file fails to parse, the service logs an error, keeps its current settings and drops the request. A stale request found at service start-up is removed.
 
== Running the Desktop Beside a Service ==
 
A service holds an operating system lock on <code>xml/config/service.lock</code> for as long as it runs. A desktop window started in the same installation folder detects it within a couple of seconds and:
 
* shows a banner "A service is running on this machine - transfers and the web log viewer are its job. Apply hands it the new settings.";
* disables its Run button, and if transfers were running from the window, pauses them at the end of the current pass;
* leaves the web log viewer to the service.
 
When the service stops, the banner clears and the window's Run button and viewer come back. The browse buttons and safe file buttons work regardless, as they open their own short-lived sessions. The lock file is left behind when the service stops; this is normal.
 
== Web Log Viewer ==
 
When Web Log Viewer is ticked on the SFTP tab, the program serves its log on <code>http://<host>:<port>/</code> (default port 8080) in both desktop and service mode. The page shows the same merged Put, Get and System log as the desktop, with a Log Level selector (remembered by the browser), System / Put / Get check boxes, the thread states, a text filter, auto-scroll and Clear. It updates live and recovers the backlog after a reconnect.
 
Read-only endpoints for monitoring tools:
 
{| class="wikitable"
|-
! URL !! Returns
|-
| <code>/</code> || The log page.
|-
| <code>/api/status</code> || JSON with version, title and the Put and Get thread states.
|-
| <code>/api/log?dest=put&#124;get&#124;sys</code> || JSON backlog for one log, optionally <code>&after=<seq></code>.
|-
| <code>/events</code> || Server-sent event stream of new log lines.
|}
 
The viewer has no login and listens on every network interface. Log lines include the SFTP host, username and file names, so keep the port on the LAN or behind a firewall. If the port is in use the program logs a warning and carries on transferring; a service retries the port every few seconds.
 
== Email Notifications ==
 
Email is off unless Email Notifications is ticked on the SFTP tab and the SMTP properties are filled in on the Email Properties tab. Messages are routed through three distribution lists:
 
{| class="wikitable"
|-
! List !! Receives !! Shipped default
|-
| '''Monitor''' || Started and Stopped (service start and shutdown, Run and Pause in the desktop) and the Daily Statistics Email (files and bytes sent and received, failures, errors raised and emailed). || Enabled, frequency 0
|-
| '''Error''' || One message per error event: Connection failed, Connection lost, Upload failed, Download failed, Local folder not found, Remote folder could not be listed, Safe file missing, Local delete failed, Remote delete failed, Backup copy failed. Connection failed is sent once per outage, however many polls fail, and a Connection lost (a session dropped part-way through a pass) opens an outage in the same way. Either is followed by a single Connection restored message (with the time the outage started and its length) when the next connection succeeds. A folder listing that failed because the session dropped is reported as Connection lost alone, not also as Remote folder could not be listed. || Enabled, frequency 15 minutes
|-
| '''Transfer''' || One message per pass that transferred at least one file, listing the files and sizes (Put Files sent, Get Files received). Passes with nothing to do send nothing; failures go to the Error list. || Disabled, frequency 0
|}
 
Subjects follow the pattern <code>SFTP Transfer <version> [Put|Get] <Event> for [<title>] on <host></code>. The body is an HTML table of label and value rows (Time, Host, Title, Folder, File, Reason, Files and so on; paths are shown in a fixed-width font) followed by a blank gap so that a corporate disclaimer appended by the mail server sits clear of the detail. The Time row is in UTC. The Frequency of a list suppresses a repeat of the same subject within that many minutes, so a file that fails on every poll produces one email per interval rather than one per poll; a suppressed email is noted in the log. A connection that stays down produces one Connection failed email for the whole outage, whatever the Frequency, and one Connection restored email when the link is back; every failed poll in between is logged as an error with a note that the email has already been sent. Because the Transfer list's subject is the same for every pass, '''its frequency must stay at 0''' or whole passes would be suppressed as repeats.
 
The Send Test Email button on the Email Properties tab sends a message to a chosen list using the settings on screen and reports the result on the status line, without needing Email Notifications to be ticked.
 
Each distribution list has an Enable flag and a comma separated address list. An installation upgraded from an earlier version gains the Transfer list, disabled, the first time the desktop is opened.


== Installing as a Windows Service ==
== Installing as a Windows Service ==


The native install package includes <code>sftpTransfer_Service.exe</code>, a service-mode launcher built with install4j. This executable manages the Windows service directly — no additional wrapper is required.
The Windows install package includes <code>sftpTransfer_Service.exe</code>, a service-mode launcher built with install4j. This executable manages the Windows service directly; no additional wrapper is required.


All service commands must be run from an elevated command prompt (Run as Administrator).
All service commands must be run from an elevated command prompt (Run as Administrator).


The default Windows service name is '''sftpTransfer_Service''' (the launcher name defined in the installer). You can specify a different name as an optional second parameter — useful if you need to run multiple instances on the same machine. If a custom name is used at install time, that same name must be passed to every subsequent command.
The default Windows service name is '''sftpTransfer_Service'''. You can specify a different name as an optional second parameter, which allows several instances to run from separate installation folders on the same machine. If a custom name is used at install time, the same name must be passed to every subsequent command.


{| class="wikitable"
{| class="wikitable"
Line 150: Line 434:
Once installed, the service also appears in the Windows Services management console (<code>services.msc</code>).
Once installed, the service also appears in the Windows Services management console (<code>services.msc</code>).


=== Testing Before Installing as a Service ===
Before installing the service, configure and test with the desktop launcher <code>sftpTransfer.exe</code> (threads start paused) or <code>sftpTransfer_Console.exe</code> (service logic with the log in a console window). Once the service is installed, the desktop can still be opened alongside it to change settings, see [[#Running the Desktop Beside a Service|Running the Desktop Beside a Service]].
 
The native install package also provides two alternatives for testing:
 
{| class="wikitable"
|-
! Launcher !! Mode !! Use case
|-
| <code>sftpTransfer_Console.exe</code> || Console (headless) || Runs the service logic in a visible console window — useful for watching log output during initial configuration
|-
| <code>sftpTransfer.exe</code> || Desktop GUI || Opens the GUI window in Start desktop mode — transfer threads start paused, allowing configuration to be verified before enabling transfers
|}


== Authentication ==
== Authentication ==


SFTPTransfer supports two authentication methods, configured per profile:
Authentication is configured once on the SFTP tab and used by both directions:


{| class="wikitable"
{| class="wikitable"
Line 171: Line 444:
! Method !! How to configure
! Method !! How to configure
|-
|-
| Password || Set the <code>Password</code> field in the profile to an AES-encrypted value. Leave the private key path empty.
| User password || Authentication Type <code>user password</code>. Enter the SFTP Username and set the SFTP Password with its Assign button.
|-
|-
| SSH private key || Set the <code>Private key path</code> field to the path of a PEM-format private key file. The password field is ignored.
| User public key || Authentication Type <code>user public key</code>. Point Private Key File and Public Key File at an existing pair, or press Create to generate a pair and then install the public key on the server yourself. If the private key is passphrase protected, set the passphrase with the Private Key Password Assign button.
|}
|}


Both methods use the JSch 2.x SSH library with Bouncy Castle as the cryptographic provider.
Host keys are checked against the Known Hosts File. With Add to Known Hosts File ticked (or <code>StrictHostKeyChecking</code> set to <code>no</code> in the jsch properties, as shipped) a new server's key is recorded on first connection.
 
All SSH is handled by the JSch library with Bouncy Castle as the cryptographic provider.


=== Password Encryption ===
=== Password Encryption ===


Passwords stored in configuration files are AES-encrypted. Plain-text passwords are never stored on disk. To encrypt a password, use the password utility included with Commander4j (the same utility used for database passwords in Commander4j's own configuration).
The SFTP password, the private key passphrase, the SMTP password and the settings password are stored encrypted in the XML files. They are set through the desktop window: each password field has an Assign button that opens a dialog, and the encrypted value is written when settings are applied. Plain text passwords are never stored. The encrypted values are specific to SFTPTransfer and cannot be copied from the configuration of other Commander4j programs.
 
== File Transfer Behaviour ==
 
=== PUT (Upload) ===
 
# SFTPTransfer polls the local input directory every 10 seconds (configurable).
# Files matching the file mask are selected for upload.
# Each file is uploaded to the remote path using a temporary name with a <code>.tmp</code> extension.
# On successful transfer, the remote file is renamed to its final name (atomic rename — the remote system never sees a partial file).
# The local source file is moved to the backup directory.
# If the transfer fails, the file is moved to the error directory and an error is logged.
 
=== GET (Download) ===
 
# SFTPTransfer lists the remote directory every 10 seconds (configurable).
# Files matching the file mask are selected for download.
# Each file is downloaded to the local path using a <code>.tmp</code> extension.
# On successful download, the local file is renamed to its final name.
# If configured, the remote file is deleted after download.
# A backup copy is optionally saved to the local backup directory.


== Thread Architecture ==
== Thread Architecture ==
When running, SFTPTransfer maintains the following threads:


{| class="wikitable"
{| class="wikitable"
Line 210: Line 463:
! Thread !! Purpose
! Thread !! Purpose
|-
|-
| TransferPUT || One thread per enabled PUT profile, each managing its own SFTP connection.
| TransferPUT || Polls the local folder and uploads. Opens its own SFTP session for each pass and closes it afterwards.
|-
| TransferGET || Polls the remote folder and downloads. Opens its own SFTP session for each pass and closes it afterwards, so Put and Get can fail and reconnect independently.
|-
|-
| TransferGET || One thread per enabled GET profile, each managing its own SFTP connection.
| EmailThread || Sends queued emails asynchronously so a slow SMTP server never delays a transfer.
|-
|-
| EmailThread || Sends error notification emails asynchronously.
| ArchiveThread || Purges backup copies older than the retention period.
|-
|-
| ArchiveThread || Periodically purges backup files older than the configured retention period.
| StatsThread || Sends the Daily Statistics Email at the configured time.
|-
| ReloadWatcher || Service mode only. Watches for <code>reload.request</code>.
|}
|}


== Hot Configuration Reload ==
In the desktop the window's own browse and safe file buttons use a separate short-lived session.
 
SFTPTransfer supports a configuration reload mode (<code>Mode_CONFIG_UPDATE</code>). When this mode is triggered, running threads re-read their configuration files without requiring a full service restart. This allows profile changes — such as updating a remote path or rotating credentials — to be applied with minimal disruption.


== Logging ==
== Logging ==


All activity is written to the Log4j log file. The log records:
Every line shown in the window is also written to <code>logs/sftpTransfer.log</code>, in both desktop and service mode, tagged <code>[PUT]</code>, <code>[GET]</code>, <code>[SYS]</code> or <code>[GUI]</code> (the window's own connections). The log records:


* Service startup and configuration loaded
* Service or window start and stop, and the settings loaded or applied
* Each thread starting and its profile settings
* Connecting, Connected (with the server's SSH banner), Disconnected and Connection lost for each session
* Every file detected, transfer attempted, and outcome
* Every command sent to the server (<code>ls</code>, <code>stat</code>, <code>put</code>, <code>get</code>, <code>rename</code>, <code>rm</code>, <code>mkdir</code>, <code>rmdir</code>, <code>setstat</code>), with the full paths on failure
* Every remote connection attempt and disconnect
* Folder headings, Sending / Sent, Receiving / Received with sizes and durations, stability retries and skips, sync deletions and the pass summary with its outcome
* File move operations (to backup or error directories)
* Safe file warnings, reload requests, service lock and web viewer events
* Errors and exceptions with full context
* Errors and exceptions with their reason


Log rotation and retention are configured in the Log4j configuration file included in the distribution.
The shipped <code>log4j2.xml</code> rolls the file over at 10 MB or daily, keeps 14 days of compressed archives and logs at <code>debug</code>. The listing of the remote folders on an idle poll is written at <code>trace</code>; set the root level to <code>trace</code> to see it in the file.


== Dependencies ==
== Dependencies ==
Line 242: Line 497:
! Library !! Version !! Purpose
! Library !! Version !! Purpose
|-
|-
| JSch || 2.27.9 || SSH/SFTP client
| JSch || 2.27.9 || SSH / SFTP client
|-
|-
| Bouncy Castle || current || Cryptographic provider for JSch
| Bouncy Castle || 1.86 || Cryptographic provider for JSch
|-
|-
| Apache Commons IO || current || WildcardFileFilter for local file matching
| Apache Commons IO || 2.21.0 || Wildcard file matching
|-
|-
| Log4j || 2.25.3 || Logging
| Jakarta Mail / Angus Mail || 2.1.5 / 2.0.5 || Email notifications
|-
| Log4j || 2.25.5 || Logging
|}
|}


See also: [[LabelServer4j]], [[Middleware4j]], [[Production Lines & Labellers]]
Source code and installers: https://github.com/C4J/sftpTransfer
 
See also: [[Middleware4j]], [[LabelServer4j]], [[Production Lines & Labellers]]


[[Category:Commander4j]]
[[Category:Commander4j]]

Latest revision as of 11:53, 22 October 2026

SFTPTransfer is a general purpose file transfer utility that moves or synchronises files between a local filesystem and a remote SFTP server. It is a standalone Java application with no dependency on any other Commander4j component: it can be used on its own wherever files need to be exchanged with an SFTP server on a schedule, and it runs on any platform with a Java runtime. It runs either as a headless background service or as a desktop application with a settings window and live log, and it uploads (Put) and downloads (Get) in independent threads.

SFTPTransfer replaces the earlier SFTPSend and SFTPGet tools with a single configurable service. Sites running Commander4j or Middleware4j typically use it to exchange files with ERP systems, customer portals, suppliers and print-and-apply labellers, but nothing in the program assumes those uses.

This page describes version 6.19.

Purpose

SFTPTransfer polls a local folder (Put) and a remote folder (Get) at a configurable interval and transfers any file that matches a file mask. Each direction works in one of two modes:

Mode Behaviour
Move Every matching file is copied to the other side and the source copy is deleted once the copy has been renamed into place. This is the default (Sync Mode unticked). Put can keep a local backup copy of each file it sends.
Sync One-way mirror. New or changed files are copied, the source is left in place and the modified time is carried across so the file is not copied again. Optionally, files and folders that have disappeared from the source are deleted from the destination.

It handles authentication, retry, backup, archiving, logging and email notification without operator involvement.

Startup Modes

SFTPTransfer supports two startup modes selected by the first command line argument:

Mode Argument Behaviour
Desktop desktop Opens the settings window. Transfer threads start paused; press Run in the toolbar to begin polling. Used to configure the program and to watch transfers.
Service service Headless. Transfer threads start immediately and run until the process is stopped. Used for production deployment as a background service.

The program reads its configuration from ./xml/config relative to the working directory, so it must be started from its installation folder. To run it by hand:

java -Dlog4j.configurationFile=./xml/config/log4j2.xml -jar sftpTransfer.jar desktop
java -Dlog4j.configurationFile=./xml/config/log4j2.xml -jar sftpTransfer.jar service

The native install packages provide launchers for each mode:

Launcher Mode Use
sftpTransfer Desktop Settings window and live log.
sftpTransfer_Console Service, in a console window Runs the service logic with the log visible on screen. Useful during initial configuration.
sftpTransfer_Service Service On Windows this executable registers and controls the Windows service, see Installing as a Windows Service. On Linux and macOS use systemd, launchd or nohup with the service argument.

A desktop window and a service can run on the same machine at the same time, see Running the Desktop Beside a Service.

Installation Folder

Folder or file Contents
xml/config/ The configuration files described below, plus the runtime files reload.request and service.lock.
xml/config/examples/ Neutral example copies of every configuration file.
ssh/ known_hosts and, if key authentication is used, the private and public key files.
safety/ The template local.safe and remote.safe files copied by the Create buttons, see Safe Files.
logs/ sftpTransfer.log and its rolled-over archives.
send/, receive/, backups/ Default local, download and backup folders used by the example configuration. Any folder can be configured instead.

Desktop Window

The window has the settings tabs on the left and a combined log on the right, separated by a draggable divider.

Tabs

Tab Settings
SFTP Title, SFTP host and port, known hosts handling, authentication type, SFTP username and password, private and public key files, email, daily statistics and web log viewer switches, settings password. Saved in sftp_common.xml.
Put Properties Upload settings. Saved in sftp_put.xml.
Get Properties Download settings. Saved in sftp_get.xml.
jsch Properties Advanced SSH session parameters passed to the JSch library. Saved in jsch_properties.xml.
Email Properties SMTP properties and the three distribution lists, plus a Send Test Email button. Saved in email_properties.xml and email_distribution_properties.xml.

Toolbar

Button Action
Run / Pause Starts or pauses the transfer threads of this window. A pass that is under way stops at the next file boundary: the file in flight is finished, the remaining files wait for the next poll, and the pass summary and Transfer email record that the pass was paused. Pausing closes the SFTP sessions. Disabled while a service is running on the same machine.
Apply Saves the settings on screen to xml/config, hands them to the running threads and writes a reload request for any service running on this machine. Enabled only when the form differs from what is on disk.
Reload Re-reads the saved files into the form, discarding edits that have not been applied.
Help Opens this page in the default browser.
About and Licences Version and third party licence information.

Settings are taken up by each thread between passes: a transfer pass that is in progress finishes with the old settings, including any sync deletions, and the status line shows "Waiting for the current transfer to complete" until both threads have changed over. Only Pause and a service stop interrupt a pass; Apply never does. Every Apply drops the SFTP sessions and reconnects with the new settings on the next pass.

The Local Folder, Remote Folder and Backup Folder fields each have a browse button. The remote browse connects with the SFTP settings as typed on the SFTP tab, so the paths can be chosen before anything has been applied or run.

Log Panel

The log panel shows the Put, Get and System logs merged in time order, each line prefixed [PUT], [GET] or [SYS]. Long lines wrap. The strip above the log holds:

Control Action
System / Put / Get Hide or show each log. Hidden lines are kept and reappear when the box is ticked again. All three are ticked at every start.
Clear Empties the log window. The log file and the web viewer are not affected.
Save Writes the lines currently shown to a text file (default logs/sftpTransfer_window_<date>.txt).
Log Level Errors only, Warnings and errors, Normal or Debug. Applies to new lines. Normal hides the polling of the remote folders; Debug shows every ls. The choice is remembered in sftp_common.xml. A running service keeps its level until the next Apply.

Two progress bars beneath the strip, Put (orange) and Get (blue), show the file being transferred, the percentage and the bytes so far. A finished file stays on show until the next one starts.

Configuration Files

All configuration is held in XML files under xml/config/. Every value has an encrypted attribute; passwords are stored encrypted and are set from the desktop window, never typed into the file. Settings added in later versions are optional: a file written by an earlier version is read with the defaults shown below.

sftp_common.xml

The connection and the program-wide switches, shared by Put and Get:

Setting Description
Title Free text shown in the window title, the web page and email subjects.
Host Address, Port SFTP server and port (default 22). There is one server: Put and Get both use it.
Known Hosts File, Add to Known Hosts File Path to the known_hosts file (default ./ssh/known_hosts) and whether an unknown host key is added automatically.
Authentication Type user password or user public key.
SFTP Username, SFTP Password SFTP login. The password is set with the Assign SFTP user password button and stored encrypted.
Private Key File, Public Key File, Key Comment, Private Key Password Key pair for public key authentication (default ./ssh/sftpTransfer and .pub). The Create button generates a new pair. The key password is set with its Assign button and stored encrypted.
Email Notifications Master switch for all email, see Email Notifications. Default off.
Daily Statistics Email, at Sends the day's transfer counts to the Monitor list once a day at the given time (HH:mm, UTC, not this machine's clock, so servers in different time zones report on the same day boundary). Default off, 06:00.
Web Log Viewer, Port Serves the logs as a read-only web page, see Web Log Viewer. Default off, port 8080.
SettingsPassword Optional. When set, the desktop window asks for it at start-up before the settings can be seen or changed. Set and cleared with its own buttons.
Log Level The desktop log level, written whenever the combo is changed.

sftp_put.xml

The upload (Put) settings:

Setting Description
Enabled Whether the Put thread transfers anything. A disabled side logs a warning at start-up and does nothing.
Title Free text, shown in the Put log and emails.
Local Folder Folder watched for outbound files. Must contain local.safe.
Local File Mask One or more wildcard masks separated by ;, for example *.xml;*.csv. See File Masks.
Backup Folder, Backup Retention Move mode only. Each sent file is copied here (sub folder structure preserved) before the local copy is deleted, and copies older than the retention period in days are purged. Greyed out in Sync Mode. The purge only runs when a marker file named backup.folder exists in the backup folder.
Poll Frequency Seconds between passes, 1 to 86400 (one day). A value outside that range is refused by Apply with a message naming the field. Default 10.
Remote Folder Destination folder on the server. Must contain remote.safe.
Temporary File Extension Files are uploaded under this extension (default .tmp) and renamed on completion, so the remote side never sees a partial file. Files carrying the extension are never selected for transfer.
Include Sub Folders Also send files from folders beneath the local folder, recreating the structure on the server. Default off.
Sync Mode Copy new or changed files and leave the local files in place. Unticked means move mode. Default off.
Overwrite Newer Sync only. Also replace a newer remote file with an older local one. Default on.
Sync Deletions Sync only. Also delete remote files which are no longer in the local folder. Default off.
Sync Folder Deletions Sync only, needs Include Sub Folders and Sync Deletions. Also remove remote folders which are no longer in the local folder. Default off.
Stability Check, Seconds, Retries Only transfer a file once it has stopped changing for the given number of seconds. A file that is still changing after the wait is checked again, up to Retries times, each after another full wait; only then is it skipped until the next poll. Retries 0 restores the single check. Default off, 5 seconds, 5 retries (an installation upgraded from an earlier version gets 5). With the defaults one file that keeps growing can hold a pass for up to six times the stability seconds.

sftp_get.xml

The download (Get) settings have the same options seen from the other side:

Setting Description
Enabled Whether the Get thread transfers anything.
Title Free text, shown in the Get log and emails.
Remote Folder Folder on the server polled for incoming files. Must contain remote.safe.
Remote File Mask One or more wildcard masks separated by ;, matched against the remote listing.
Local Folder Folder where downloaded files are saved. Must contain local.safe.
Poll Frequency Seconds between passes, 1 to 86400 (one day). A value outside that range is refused by Apply with a message naming the field. Default 10.
Temporary File Extension Files are downloaded under this extension and renamed on completion.
Include Sub Folders Also fetch files from folders beneath the remote folder.
Sync Mode Copy new or changed files and leave the remote files in place. Unticked means move mode (the remote file is deleted after a completed download).
Overwrite Newer Sync only. Also replace a newer local file with an older remote one. Default on.
Sync Deletions Sync only. Also delete local files which are no longer in the remote folder.
Sync Folder Deletions Sync only. Also remove local folders which are no longer in the remote folder.
Stability Check, Seconds, Retries Only fetch a file once the remote listing shows it unchanged for the given number of seconds. A file that is still changing is listed again, up to Retries times, each after another full wait, before it is skipped until the next poll. Retries 0 restores the single check. Default off, 5 seconds, 5 retries.

jsch_properties.xml

Session parameters passed directly to the JSch library: strict host key checking, preferred authentication methods, key exchange, cipher, MAC and host key algorithm lists, compression and proxy settings. Each row can be enabled or disabled and encrypted individually. The shipped defaults suit most servers; change them only when a server insists on a particular algorithm.

email_properties.xml

Jakarta Mail SMTP properties. Six rows are always present and cannot be removed: mail.smtp.host, mail.smtp.port, mail.smtp.from, mail.smtp.auth, mail.smtp.user and mail.smtp.password; a file that lacks any of them gains the row, disabled and empty, the next time the desktop is opened. The Add Property button beside Send Test Email offers the usual TLS, SSL, socket factory, timeout, charset and debug properties with suggested values, or any other name typed in; Remove Property deletes the selected row and refuses the six required ones. The password row is stored encrypted. Rows with Enable unticked are left out of the session. A server that answers 530 ... anonymous mail or a similar authentication error usually needs mail.smtp.auth enabled and set to true.

email_distribution_properties.xml

The three distribution lists. Each has an address list (comma separated), an Enable flag and a Frequency in minutes, see Email Notifications.

log4j2.xml

Log file location, rollover and retention, see Logging.

Safe Files

A transfer pass does nothing unless both of these files exist:

File Where
local.safe In the root of the Local Folder (Put and Get each have their own).
remote.safe In the root of the Remote Folder on the server.

The files prove that the configured folders are the intended ones and that any network share or mount behind them is actually present. Without them a mistyped path, an unmounted share or an empty folder could otherwise look like "nothing to send" or, with Sync Deletions on, empty the other side. Only the name matters, not the content.

  • The Put and Get tabs have Create (tick) and Remove (cross) buttons beside the Local Folder and Remote Folder fields. Create copies the template from the safety folder into the folder typed in the field, after showing the full path and asking for confirmation. The remote buttons open their own SFTP session.
  • A missing safe file is logged once as a warning (Nothing sent, safe file not found : ...) and again after Apply or a restart; transfers resume at the next poll once the file appears.
  • The safe files are never transferred, deleted or counted, whatever the file mask.
  • The remote check lists the remote folder, so the SFTP account must be permitted to list it.
  • Create the local file only while any share it lives on is mounted, otherwise the empty mount point is what gets approved.

When upgrading from a version before 6.10, nothing transfers until the safe files have been created. Where the remote server is owned by another party they will need to place remote.safe for you.

File Masks

The Local File Mask (Put) and Remote File Mask (Get) accept the wildcards * and ?. Several masks can be given separated by ;, for example *.xml;*.csv; spaces around a mask are ignored.

  • Put matching is case-insensitive; Get matching follows the server listing and is case-sensitive.
  • A blank mask matches nothing on both sides.
  • Files carrying the Temporary File Extension are always skipped, even with a mask of *.
  • A folder whose name matches the mask is skipped, never transferred or deleted.
  • The mask is also the scope of Sync Deletions: only files matching it are deleted from the destination.

Transfer Behaviour

Move Mode

Move mode is in force when Sync Mode is unticked.

  1. Each poll collects the files matching the mask (and, with Include Sub Folders, those in folders beneath).
  2. With Stability Check on, the thread waits until the collection is the configured number of seconds old, then re-checks each file's size and modified time. A file that has changed is waited for again, up to the configured number of Retries (logged as Still changing ... retry n of m); a file that is still changing after the last retry is skipped with a warning and picked up on the next poll.
  3. Put: the file is copied to the backup folder if backup is on, uploaded under the temporary extension, renamed to its final name, and only then deleted locally. Get: the file is downloaded under the temporary extension, renamed, and only then deleted from the server. A partial transfer is removed and the source is left in place.
  4. Sub folders are created on the destination as needed. The configured folders themselves are never created or removed, and emptied source sub folders are left in place.
  5. A file that keeps failing is retried and logged on every poll. There is no error folder.

Sync Mode

Sync Mode is a one-way mirror from source to destination; the source is never deleted. For each file the destination copy is compared by size and modified time:

Destination copy Action
Missing, older, or same time with a different size Transferred. The modified time is carried across.
Identical (same size and time) Skipped silently.
Newer than the source Transferred only when Overwrite Newer is ticked. Otherwise kept, logged at Debug level only.

With Include Sub Folders on, empty folders are created on the destination too.

Sync Deletions: at the end of each folder, a destination file is deleted when it matches the mask, does not carry the temporary extension and no file of that name exists in the matching source folder. Folders are never removed by this option, and folders that exist only at the destination are left alone with their contents. Nothing is deleted when a folder cannot be listed or when the pass is interrupted. An empty source folder (with its safe file present) empties the destination of mask-matching files.

Sync Folder Deletions (needs Include Sub Folders and Sync Deletions): a folder that exists only at the destination has its mask-matching files deleted and is removed once it is empty. A folder still holding a file the mask does not cover is kept with that file. A folder is only judged to be destination-only when nothing of that name exists at the source, including a file or a link.

Each folder is processed in full (transfers, then deletions) before the next one is started, and a log line Folder <path> is written before the first action in a folder.

Points to note:

  • Do not tick Sync Deletions on both Put and Get against the same pair of folders.
  • Sync Folder Deletions treats every folder beneath the destination as in scope. A "processed" folder created on the server by another system, or an archive folder beneath the receive folder, will lose its mask-matching files.
  • On a Mac, Finder's .DS_Store file does not match a mask such as *.xml, so a browsed folder is emptied but not removed.
  • Backup is for move mode: in Sync Mode the source file is still there, so the backup row is greyed out and no backup copy is taken.

Pass Summary

Every pass that transferred, failed or skipped at least one file ends with Transfer finished - n sent, n failed, n skipped in n s. A pass that does not run to its end says instead Transfer interrupted by shutdown, Transfer paused from the window or Transfer abandoned, connection lost, followed by the same counts, and the Transfer email for that pass carries an Outcome row saying so and listing the files that did get through; the rest are sent or fetched on the next poll. A pass with nothing to do writes nothing at the Normal log level, so an idle service produces a quiet log.

Each thread disconnects from the server at the end of every pass and reconnects at the start of the next, so a long poll interval never leaves an idle session for a firewall or the server to drop. The log therefore shows Connecting, Connected and Disconnected on every pass, including an idle one.

Applying Settings to a Running Service

A service reads xml/config at start-up and whenever the file xml/config/reload.request appears. The desktop's Apply button writes the settings and then writes that file; a service on the same machine collects it within a second, re-reads the six configuration files, hands the settings to its threads (between passes) and deletes the file. The desktop status line reports whether a service collected the request.

An empty reload.request created by any other means works too, so settings prepared on another machine can be copied into place and activated without restarting the service. If any configuration file fails to parse, the service logs an error, keeps its current settings and drops the request. A stale request found at service start-up is removed.

Running the Desktop Beside a Service

A service holds an operating system lock on xml/config/service.lock for as long as it runs. A desktop window started in the same installation folder detects it within a couple of seconds and:

  • shows a banner "A service is running on this machine - transfers and the web log viewer are its job. Apply hands it the new settings.";
  • disables its Run button, and if transfers were running from the window, pauses them at the end of the current pass;
  • leaves the web log viewer to the service.

When the service stops, the banner clears and the window's Run button and viewer come back. The browse buttons and safe file buttons work regardless, as they open their own short-lived sessions. The lock file is left behind when the service stops; this is normal.

Web Log Viewer

When Web Log Viewer is ticked on the SFTP tab, the program serves its log on http://<host>:<port>/ (default port 8080) in both desktop and service mode. The page shows the same merged Put, Get and System log as the desktop, with a Log Level selector (remembered by the browser), System / Put / Get check boxes, the thread states, a text filter, auto-scroll and Clear. It updates live and recovers the backlog after a reconnect.

Read-only endpoints for monitoring tools:

URL Returns
/ The log page.
/api/status JSON with version, title and the Put and Get thread states.
/api/log?dest=put|get|sys JSON backlog for one log, optionally &after=<seq>.
/events Server-sent event stream of new log lines.

The viewer has no login and listens on every network interface. Log lines include the SFTP host, username and file names, so keep the port on the LAN or behind a firewall. If the port is in use the program logs a warning and carries on transferring; a service retries the port every few seconds.

Email Notifications

Email is off unless Email Notifications is ticked on the SFTP tab and the SMTP properties are filled in on the Email Properties tab. Messages are routed through three distribution lists:

List Receives Shipped default
Monitor Started and Stopped (service start and shutdown, Run and Pause in the desktop) and the Daily Statistics Email (files and bytes sent and received, failures, errors raised and emailed). Enabled, frequency 0
Error One message per error event: Connection failed, Connection lost, Upload failed, Download failed, Local folder not found, Remote folder could not be listed, Safe file missing, Local delete failed, Remote delete failed, Backup copy failed. Connection failed is sent once per outage, however many polls fail, and a Connection lost (a session dropped part-way through a pass) opens an outage in the same way. Either is followed by a single Connection restored message (with the time the outage started and its length) when the next connection succeeds. A folder listing that failed because the session dropped is reported as Connection lost alone, not also as Remote folder could not be listed. Enabled, frequency 15 minutes
Transfer One message per pass that transferred at least one file, listing the files and sizes (Put Files sent, Get Files received). Passes with nothing to do send nothing; failures go to the Error list. Disabled, frequency 0

Subjects follow the pattern SFTP Transfer <version> [Put|Get] <Event> for [<title>] on <host>. The body is an HTML table of label and value rows (Time, Host, Title, Folder, File, Reason, Files and so on; paths are shown in a fixed-width font) followed by a blank gap so that a corporate disclaimer appended by the mail server sits clear of the detail. The Time row is in UTC. The Frequency of a list suppresses a repeat of the same subject within that many minutes, so a file that fails on every poll produces one email per interval rather than one per poll; a suppressed email is noted in the log. A connection that stays down produces one Connection failed email for the whole outage, whatever the Frequency, and one Connection restored email when the link is back; every failed poll in between is logged as an error with a note that the email has already been sent. Because the Transfer list's subject is the same for every pass, its frequency must stay at 0 or whole passes would be suppressed as repeats.

The Send Test Email button on the Email Properties tab sends a message to a chosen list using the settings on screen and reports the result on the status line, without needing Email Notifications to be ticked.

Each distribution list has an Enable flag and a comma separated address list. An installation upgraded from an earlier version gains the Transfer list, disabled, the first time the desktop is opened.

Installing as a Windows Service

The Windows install package includes sftpTransfer_Service.exe, a service-mode launcher built with install4j. This executable manages the Windows service directly; no additional wrapper is required.

All service commands must be run from an elevated command prompt (Run as Administrator).

The default Windows service name is sftpTransfer_Service. You can specify a different name as an optional second parameter, which allows several instances to run from separate installation folders on the same machine. If a custom name is used at install time, the same name must be passed to every subsequent command.

Command Effect
sftpTransfer_Service.exe /install Register as sftpTransfer_Service with automatic startup on boot
sftpTransfer_Service.exe /install "My SFTP Service" Register with a custom service name
sftpTransfer_Service.exe /install-demand Register with manual startup only
sftpTransfer_Service.exe /start Start the service
sftpTransfer_Service.exe /stop Stop the service (waits for graceful shutdown)
sftpTransfer_Service.exe /restart Restart the service
sftpTransfer_Service.exe /status Query status (exit code: 0 = running, 3 = stopped)
sftpTransfer_Service.exe /uninstall Remove the service registration

If a custom service name was used at install time, append it to all subsequent commands:

sftpTransfer_Service.exe /start "My SFTP Service"
sftpTransfer_Service.exe /stop "My SFTP Service"
sftpTransfer_Service.exe /uninstall "My SFTP Service"

Once installed, the service also appears in the Windows Services management console (services.msc).

Before installing the service, configure and test with the desktop launcher sftpTransfer.exe (threads start paused) or sftpTransfer_Console.exe (service logic with the log in a console window). Once the service is installed, the desktop can still be opened alongside it to change settings, see Running the Desktop Beside a Service.

Authentication

Authentication is configured once on the SFTP tab and used by both directions:

Method How to configure
User password Authentication Type user password. Enter the SFTP Username and set the SFTP Password with its Assign button.
User public key Authentication Type user public key. Point Private Key File and Public Key File at an existing pair, or press Create to generate a pair and then install the public key on the server yourself. If the private key is passphrase protected, set the passphrase with the Private Key Password Assign button.

Host keys are checked against the Known Hosts File. With Add to Known Hosts File ticked (or StrictHostKeyChecking set to no in the jsch properties, as shipped) a new server's key is recorded on first connection.

All SSH is handled by the JSch library with Bouncy Castle as the cryptographic provider.

Password Encryption

The SFTP password, the private key passphrase, the SMTP password and the settings password are stored encrypted in the XML files. They are set through the desktop window: each password field has an Assign button that opens a dialog, and the encrypted value is written when settings are applied. Plain text passwords are never stored. The encrypted values are specific to SFTPTransfer and cannot be copied from the configuration of other Commander4j programs.

Thread Architecture

Thread Purpose
TransferPUT Polls the local folder and uploads. Opens its own SFTP session for each pass and closes it afterwards.
TransferGET Polls the remote folder and downloads. Opens its own SFTP session for each pass and closes it afterwards, so Put and Get can fail and reconnect independently.
EmailThread Sends queued emails asynchronously so a slow SMTP server never delays a transfer.
ArchiveThread Purges backup copies older than the retention period.
StatsThread Sends the Daily Statistics Email at the configured time.
ReloadWatcher Service mode only. Watches for reload.request.

In the desktop the window's own browse and safe file buttons use a separate short-lived session.

Logging

Every line shown in the window is also written to logs/sftpTransfer.log, in both desktop and service mode, tagged [PUT], [GET], [SYS] or [GUI] (the window's own connections). The log records:

  • Service or window start and stop, and the settings loaded or applied
  • Connecting, Connected (with the server's SSH banner), Disconnected and Connection lost for each session
  • Every command sent to the server (ls, stat, put, get, rename, rm, mkdir, rmdir, setstat), with the full paths on failure
  • Folder headings, Sending / Sent, Receiving / Received with sizes and durations, stability retries and skips, sync deletions and the pass summary with its outcome
  • Safe file warnings, reload requests, service lock and web viewer events
  • Errors and exceptions with their reason

The shipped log4j2.xml rolls the file over at 10 MB or daily, keeps 14 days of compressed archives and logs at debug. The listing of the remote folders on an idle poll is written at trace; set the root level to trace to see it in the file.

Dependencies

Library Version Purpose
JSch 2.27.9 SSH / SFTP client
Bouncy Castle 1.86 Cryptographic provider for JSch
Apache Commons IO 2.21.0 Wildcard file matching
Jakarta Mail / Angus Mail 2.1.5 / 2.0.5 Email notifications
Log4j 2.25.5 Logging

Source code and installers: https://github.com/C4J/sftpTransfer

See also: Middleware4j, LabelServer4j, Production Lines & Labellers