Problem
My current project relies on many 3rd parties (financial institutions) and their SFTP servers. Alpakka SFTP works as-is for a good number of these, but a few of them run legacy SFTP servers that require the addition of the HostKeyAlgorithms SFTP options flag when writing files their remote server.
Due to this pull request from 2017, the option flag field in the SftpSettings case class was removed. Here is the code blob for reference.
My “work around” has just been to execute a shell script for these legacy 3rd parties:
aws s3 cp "$S3_PATH" "$FILE_NAME"
# Main command:
sftp -oHostKeyAlgorithms=+ssh-rsa $USERNAME@stm.experian.com:$REMOTE_DIR <<< "put $FILE_NAME"
# This one also works:
sshpass -p $PASSWORD sftp -oStrictHostKeyChecking=no "$USERNAME@stm.experian.com:$REMOTE_DIR" <<< "put $FILE_NAME"
but my standard attempt at implementing this in scala using Alpakka’s FTP lib doesn’t work:
final val DefaultChunkSize = 8192
val sftpSettings = SftpSettings
.create(InetAddress.getByName(config.hostName))
.withPort(config.port)
.withCredentials(credentials)
.withStrictHostKeyChecking(false)
.into(config.certificate) { (settings, cert) =>
val identity = SftpIdentity.createRawSftpIdentity(cert.getBytes)
settings.withSftpIdentity(identity)
}
// :( Doesn't work with those legacy 3rd parties, but does for others
// Listing directories and file reading works as expected for all 3rd party SFTP server providers
override def writeFile(basePath: String, fileName: String, content: Array[Byte]): Future[IOResult] = {
val path = s"$basePath/$fileName"
Source(content.toList)
.grouped(DefaultChunkSize)
.map(c => ByteString(c.toArray))
.runWith(Sftp.toPath(path, sftpSettings, true))
}
As such, are there any solutions/work arounds given the current Alpakka FTP lib? Or can we bring back the SFTP options flag to the SftpSettings case class to support such cases as above? The options flag default can be Nil as it shouldn’t need to be set in the common case.
Error message
akka.stream.IOOperationIncompleteException: IO operation was stopped unexpectedly after 0 bytes because of net.schmizz.sshj.sftp.SFTPException: Requested operation is not supported.
at akka.stream.alpakka.ftp.impl.FtpIOSinkStage$$anon$3.matFailure(FtpIOGraphStage.scala:246)
at akka.stream.alpakka.ftp.impl.FtpGraphStageLogic.preStart(FtpGraphStageLogic.scala:42)
at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:306)
at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:619)
at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:727)
at akka.stream.impl.fusing.ActorGraphInterpreter.preStart(ActorGraphInterpreter.scala:776)
at akka.actor.Actor.aroundPreStart(Actor.scala:548)
at akka.actor.Actor.aroundPreStart$(Actor.scala:548)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart(ActorGraphInterpreter.scala:716)
at akka.actor.ActorCell.create(ActorCell.scala:644)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:514)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:536)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295)
at akka.dispatch.Mailbox.run(Mailbox.scala:230)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: net.schmizz.sshj.sftp.SFTPException: Requested operation is not supported.
at net.schmizz.sshj.sftp.Response.error(Response.java:140)
at net.schmizz.sshj.sftp.Response.ensurePacketTypeIs(Response.java:117)
at net.schmizz.sshj.sftp.SFTPEngine.open(SFTPEngine.java:143)
at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:68)
at net.schmizz.sshj.sftp.SFTPClient.open(SFTPClient.java:73)
at akka.stream.alpakka.ftp.impl.SftpOperations.$anonfun$storeFileOutputStream$1(SftpOperations.scala:168)
at scala.util.Try$.apply(Try.scala:213)
at akka.stream.alpakka.ftp.impl.SftpOperations.storeFileOutputStream(SftpOperations.scala:163)
at akka.stream.alpakka.ftp.impl.SftpOperations.storeFileOutputStream$(SftpOperations.scala:162)
at akka.stream.alpakka.ftp.impl.FtpLike$$anon$3.storeFileOutputStream(FtpLike.scala:73)
at akka.stream.alpakka.ftp.impl.FtpLike$$anon$3.storeFileOutputStream(FtpLike.scala:73)
at akka.stream.alpakka.ftp.impl.FtpIOSinkStage$$anon$3.doPreStart(FtpIOGraphStage.scala:238)
at akka.stream.alpakka.ftp.impl.FtpGraphStageLogic.preStart(FtpGraphStageLogic.scala:39)
... 15 common frames omitted