2010 m. kovo 17 d., trečiadienis

SFTP from Oracle

So, the goal: upload/download files from Oracle server to/from SFTP server.
Constraints: the operation must be able to be triggered from within pl/sql by an Oracle job.

As far as I know by now there are no means to implement this functionality in pure pl/sql.

However there is a number of Java libraries which provide required API. Below is an example of unilizing Jsch library.

There four main steps required:

  1. Install Jsch library into Oracle

  2. Create your API for aforesaid functionality and install it to Oracle

  3. Create pl/sql wrappers for your API

  4. Grant java permissions

  5. Enjoy results


Step 1. Install Jsch library into Oracle

Download Jsch library from here (version jsch-0.1.42.jar). At the time of reading this post the version may have been changed and the link been broken. Anyway you must be able to find downloadable jar here.

Install the downloaded jar into Oracle following these guidelines here.

Make sure that installed classes are valid. You can do it by examining result of the following query:

SELECT object_name, status FROM user_objects WHERE object_name LIKE 'com/jcraft/%';


In my particular case I had to recompile the classes to have their status VALID. Below is an example of recompilation statement for class com/jcraft/jsch/Buffer:

alter java class "com/jcraft/jsch/Buffer" resolve;

See details on alter java statement here (amidst other locations of course).

Step 2. Create your API for aforesaid functionality and install it to Oracle


  • Create a Java project in your preferred IDE (Eclipse, Netbeans, JDeveloper etc.) and add to it the downloaded jar.

  • Download Jsch source code and examples from here. If this link is broken you can find the correct link here.

  • In examples find Sftp.java and put it in your project.

  • Refactor Sftp.java so that you have two static public classes for get and put operations.


    Of course this is easer to say than to do. Below you will find the result of my prototyping efforts. You can use this as a starting point.

    But if you wish to do it all by yourself you could be interested in knowing what difficulties I faced. First of all, no Jsch java doc exist. Jsch documentation is its source code itself. However, in my particular case, Jsch classes (in the downloaded jar) were compiled in the manner precluding from examining variables at debug time. So I had to throw Jsch jar out of my java project and include source files into it.

    The next point is that Sftp.java has generically interactive architecture. That is, it requres User intervention and even imports visual library (swing, awt). Meanwihle, remember the uploads/download will be triggered by jobs and consequently can not interact with user.

    Thus, every dialog must by abandoned. Fortunately, most dialogs merely prompts for password and other connection details which we instead can preconfigure.

    One dialog however is worth some extra words: user's acceptance of host key. This dialog is triggered when the server's public key Jsch knows does not coincide with actual server's public key. So if we replace the dialog by constant negative answer the connection will fail when the keys do not coinside.

    Jsch keeps known keys in so called known hosts file. In this file Jsch keeps host urls and correspondent keys. The path to the file can by set by (for example) method jsch.setKnownHosts.

    To make the keys coinside Oracle administrator has to have a tools allowing identification of keys and saving them into the faile. However, this task goes beyond the subject of this page. Possibly, I will return to this issue afterwords.

    And one more thing not to forget. Exit channels and sessions if you do not want to restart Oracle database :)

    In my my particular case the class looks like this:

    package org.ssh;

    import java.io.BufferedWriter;
    import java.io.FileWriter;
    import java.io.IOException;

    import com.jcraft.jsch.Channel;
    import com.jcraft.jsch.ChannelSftp;
    import com.jcraft.jsch.JSch;
    import com.jcraft.jsch.JSchException;
    import com.jcraft.jsch.Session;
    import com.jcraft.jsch.SftpException;
    import com.jcraft.jsch.UserInfo;

    public class Sftp {
    private static final String PASS = "test";
    private static final String KNOWN_HOSTS = "C:\\Users\\Mike\\Documents\\KNOWN_HOSTS";

    private static ChannelSftp getChannel() throws JSchException {
    JSch jsch = new JSch();
    jsch.setKnownHosts(KNOWN_HOSTS);
    final String user = "me";
    final String host = "localhost";
    final int port = 22;
    Session session = jsch.getSession(user, host, port);
    session.setPassword(PASS);
    UserInfo ui = new MyUserInfo();
    session.setUserInfo(ui);
    session.connect();
    Channel channel = session.openChannel("sftp");
    channel.connect();
    return (ChannelSftp) channel;
    }

    private static void finalizeConnection(ChannelSftp channel, Session session) {
    channel.exit();
    session.disconnect();
    }

    public static void main(String[] arg) {
    // get("put_by_me11.txt", "c:\\ftp\\big_to_upload111.txt");
    writeFile("adasf");

    }

    public static class MyUserInfo implements UserInfo {
    public String getPassword() {
    return passwd;
    }

    public boolean promptYesNo(String str) {
    // TODO: enter sane code
    return true;
    }

    String passwd;

    public String getPassphrase() {
    return null;
    }

    public boolean promptPassphrase(String message) {
    return true;
    }

    public boolean promptPassword(String message) {
    throw new UnsupportedOperationException();
    }

    public void showMessage(String message) {
    throw new UnsupportedOperationException("No dialogs suppported");
    }
    }

    public static String get(String src, String dst) {
    return doCommand("get", src, dst);
    }

    public static String doCommand(String command, String src, String dst) {
    String errMsg = null;
    ChannelSftp c = null;
    Session s = null;
    try {
    c = getChannel();
    s = c.getSession();
    if (command.equals("get")) {
    c.get(src, dst);
    } else {
    c.put(src, dst);
    }
    } catch (SftpException e) {
    errMsg = e.toString();
    System.out.println(e.toString());
    } catch (JSchException e) {
    errMsg = e.toString();
    System.out.println(e.toString());
    } finally {
    finalizeConnection(c, s);
    }
    return errMsg == null ? "Ok" : errMsg;
    }

    public static String put(String src, String dst) {
    return doCommand("put", src, dst);
    }

    public static void writeFile(String s) {
    try {
    BufferedWriter out = new BufferedWriter(new FileWriter(
    "c:\\test.txt"));
    out.write(s);
    out.close();

    } catch (IOException e) {
    }
    }
    }




    The above implementation is only part of prototyping efforts and therefore can not be used as is.

    KNOWN_HOSTS constant, in the above source, is a path to a file which contains list of known hosts where each known host entry has a host url together with the host public key(s). You must know Jsch methods setKnownHosts accepts only stream or file but not just host url for example.

    Now, that we have the API, we have to install it to Oracle. You can do it again with loadjava or compile the java source directly in Sqlplus. The latter solution is preferrable if you expect future modification of your API. This is because, in contrast with loadjava, sqlplus requires only oracle ordinary staff to be accomplished (you do not have to copy files with your java source to Oracle server).

    Below is the statement to install and compile Java source into Oracle from within Sqlplus (or PlsqlDeveloper or TOAD):

    create or replace and compile java source named mike_sftp as
    JAVA_SOURCE
    ;


    In the statement above the JAVA_SOURCE is actually the full code of the Sftp class exposed in this blog page too. I replaced it here to keep this page brief (while this may not be possible :) ).

    Step 3. Create pl/sql wrappers for your API

    This is quite easy step :). The code is below:

    CREATE OR REPLACE FUNCTION mike_java_put(src VARCHAR, dst VARCHAR) RETURN VARCHAR AS
    LANGUAGE JAVA NAME 'org.ssh.Sftp.put(java.lang.String, java.lang.String) return java.lang.String';


    Step 4. Grant java permissions.
    This is the step required for GET operation. One has to know that for for Java installed in oracle one have grant permissions to write files. In my case the code was:

    BEGIN
    dbms_java.grant_permission('SYSTEM', 'SYS:java.net.SocketPermission', 'localhost', 'resolve');
    dbms_java.grant_permission('SYSTEM', 'SYS:java.io.FilePermission', '<<all>>', 'write');
    END;


    If you are lacking file write permissions it is quite possible that you will not obtain cleare error message from Jsch. This is because the library can not handle gracely all huge number of various errors which do not deal with SHH. Therefore, if your make your put command function correctly and the get command does not function with no clear error message then it is probably a security issue and you should grant some additional permissions.

    Step 4 Enjoy your job ;)

    DECLARE
    s_result VARCHAR(100);
    BEGIN
    s_result := mike_java_get('to_download_by_oracle.txt', 'downloaded_by_oracle.txt');
    dbms_output.put_line(s_result);
    END;
  • 2010 m. vasario 16 d., antradienis

    Green Sherpa

    Yesturday I found an interesting personal finance management online service - Green Sherpa.
    I had been looking for a PFM which would provide planning tools. And Green Sherpa turned out to be one of them.

    Despite negative view in Prosper Lending Review, I decided to give it a try:

    The promised functionality looked to be available but oops! The controls in the planning module, "Add" (planned amount) buttons in Cash Flow Projection page, did not function. It seems that Green Sherpa overlooked a bug in the module I needed mostly...

    So I'm looking further for other tools...

    2010 m. vasario 15 d., pirmadienis

    Personal Finance Management software comparison

    A couple of days ago I returned to my old plans to make use of up-to-date financial management software. Last year Microsof declare discontinueing MS Money. So, I had to research the market of the like soft.

    There two major streams in the field as described in an eHow articl by Maxwell Payne: old good desktop applications and new, most often, WEB 2.0 style, web applications available online. I chosen on-line.

    Of course wikies was the most complete source of available choices: the review below is based on its Comparison of personal financial management software.

    After filtering only free web based applications which are available internationally (at least as the wiki says) the following list remained:
    1. Budgetpulse
    2. Manubia
    3. Meniga
    4. MoneyRight
    5. moneyStrands
    6. MoneyTrackin'
    7. Onyem
    8. PennyTrac
    9. PersonalAccountingSoftware
    10. Rudder
    11. Thrive
    12. Wehuhu
    13. Wesabe
    14. Winancial
    15. Yodlee MoneyCenter
    In the wikie all they are qualified either as international or have not this attribute set at all.

    The functionality I need is
    1. Ability to have an account for a bank which is not supported for automatic transaction download. There is a big number of banks, especially this regards USA, Canada, Western Europe etc, which support this. In other parts the world however there is a bigger number of banks that DO NOT provide such service.
    2. Ability to have cash accounts.
    3. Ability to having repeating payments / planned payments.
    4. Ability to plan finances. That is ability to graphically view future states, the data being based on the registered planned payments.
    So, by its order:
    1. Budgetpulse:
    Overall the impression is good although planning finances turned to be not possible. (I'm afraid such a functionality is rare).
    Requerement #1 - passed. Req. #2 - passed. Req.3 - passed. Req. #4 - failed because charts do not include planned (repeating) transactions.

    2. Manubia:
    Since I can not make any use of it I give this system 0 points of 10. I only speak English, German, Russian and Lithuanian. Regretably Manubia is in Portuguese only. So I would not consider this system an international at all.

    3. Meniga:
    It is not international: "Meniga is still in beta and, for now, only registered users of Íslandsbanki’s online bank can sign up" was declared on its registration page.

    4. MoneyRight
    I could not even sign up due to, maybe, some bug ever invalidating my zip code.

    5. moneyStrands
    Recurring transactions are not supported.

    6. MoneyTrackin'
    No analisys tools at all.

    7. Onyem
    I have not tried this because its tour shows only analisys of the past but nothing about the future.

    8. PennyTrac
    The industrial niche of this system is not transaction tracking etc... It's more for investors...

    9. PersonalAccountingSoftware
    It turned out to be windows based software (an error in wikipedia). Wikipedia entry updated.

    10. Rudder
    Does not support cash accounts

    11. Thrive
    Does not support cash accounts

    12. Wehuhu
    Meets all the abovementioned functional requirements while the system crashed after 10 or so operations.

    13. Wesabe
    Recurring transactions are not supported

    14. Winancial
    This is single language system. You can use it neither in English.

    15. Yodlee MoneyCenter
    Has only 10 predefined currencies. Mine is absent in the list.

    Thus... there is no single suitable system. At least a free one.

    2008 m. spalio 30 d., ketvirtadienis

    Starting RichFaces with Eclipse

    Hello!

    Here what I faced when followed yesterday the "An Introduction To JBoss RichFaces", an extremely good article on "kickstart" of using RichFaces with Eclipse. I thank very much the author of the article Max Katz for the excellent tutorial!

    But, of course, I never passed through without problems... And I had problems here too...

    1 To be smart :) I installed the currently latest release of Eclipse for Java EE Developers (Ganymede: ) instead of Europa release given in the article. Then I had a happy time trying to install latest JBoss Tools plugin (currently JBossTools-2.1.2.GA-ALL-win32.zip) with the all same result: JBoss Tools failed to visualize its palette. Finally I found myself stairing at the bootom right of the JBoss Release 2.1.2.GA page where one can find that the 2.1.2.GA is only for WTP 3.3.2 (Europa release) but not Ganymede (WTP 3.4.x release). That is... the currently latest release of JBoss Tools is not compatible with the currently latest release of Eclipse. Pitty. But nothing doing though.
    So, I installed Europa and enjoyed seeing the JBoss Tools palette.

    2 Now, let us take the template souce from the article. I put it as instructed but the addition environmental choice had not appaired :( . When tried to create JSF project with JSF Environment "JSF 1.2 with Facelets" I got an error on Tomcat 6.0.18 server start:

    SEVERE: Error configuring application listener of class com.sun.faces.config.ConfigureListener
    java.lang.ClassNotFoundException: com.sun.faces.config.ConfigureListener

    I had to spend a lot of time not knowing that after we install new templates we MUST do something with the workspace directory of Eclipse. In my case, I closed Eclipse, deleted its workspace directory lanched Eclipse again and... voilà! I got the power :) ... The environmental choice did appair.

    3 So, its time to server's start... Ooops!!! It crashes again :'(. Thank you, the reason differs and I could start figureing out the reason again. The error message was: Content is not allowed in prolog. Ok, it was quite simple to resolve. In Web.xml between the xml header and web-app version number there a line appaired
    (2.5 == "2.5")
    surely, xml files may not have anything there. So I deleted this line and successfully started server. This line appaired in xml because in the project creation wizard's I selected Servlet version 2.5 instead of 2.4. If I leave the default the problem does not appair. So you must follow the Max Katz tutorial very precisely: hi writes "
    1. For a template, select one based on the Tomcat version you are using
    2. Click Finish to create the project
    "
    If you press Finish you will never go to the wrong place to choose wrong configuration as I regretably did.

    So I hope, dear reader, that you will not repeat my mistakes. :)

    Embedded JBoss on Tomcat6 with Java JDK 1.6

    Several days ago I started evaluating Seam framework (http://www.seamframework.org) . Of course I found that the best application server to use with the framework is for sure JBoss AS.
    That is, to follow the tutorial (http://docs.jboss.com/seam/latest/reference/en-US/html/tutorial.html) one has to install either JBoss AS or, if one preferes Tomcat, Embedded JBoss.
    Our customer wants Tomcat. Thus, we must go with:

    the latest Tomcat release (Tomcat 6.0.18)
    the latest Java (JDK 1.6.0_07) and
    the latest embedded jbos (embedded-jboss-beta3)
    Windows XP SP3

    In the installation guide we find that to work with JDK 1.6 a Java option must be set:
    -Dsun.lang.ClassLoader.allowArraySyntax=true
    That is perfectly clear but... don't expect to work if you install everithing as instucted in documentation: the issues is that if you install Tomcat 6 under its default path ("C:\Program Files\Apache Software Foundation\Tomcat 6.0") then the Tomcat server will crash starting!!!
    After hours of googling I found a forum thread http://www.seamframework.org/18890.lace posing this problem and solving it.
    Tomcat log shows
    ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] Error installing to Parse: name=vfsfile:/C:/Dokumente%20und%20Einstellungen/nes/Eigene%20Dateien/workspace/SUPCockpit/test-build/ state=Not Installed mode=Manual requiredState=Parse
    (the example taken from http://www.seamframework.org/18890.lace).
    And the problem is that the path to vfsfile includes spaces. Of course it does, because default tomcat installation path does!!! So, just reinstall Tomcat under installation directory with no spaces an the problem must go away! :)

    Why am I starting my weblog

    Nowadays I am performing an investigation in the field of available Rich Internet Application technologies. The targeted customer is an isurance company which would like to move its client/server intranet applicatio to Web. The programming language is one of the thing we have allready made a positive decisions about.

    I spent a lot of time looking for the frameworks and IDE. Everybody knows there is a lot of them and I in no case want to start fruitless discussions on what frameworks or IDE is better. Nevertheless, I would not object comparying them to each other as far as we consider a feature in respect to abovementioned needs.

    My main goal now is to help people who are installing a lot of different software, evaluating it, combining it with other pieces of soft and having lots of headackes, not to repeate mistakes that I did.