powershell - HDInsight Oozie: Hive Job Parameters -


I am working in the process of automating the hive jobs using Oozie scipts. In the workflow.exml, I get the value from the Powershell script file (OZ Job Script).
Where in the HCl file I am unable to get the value defined in the PowerShell script file (Ozzy Job Script).

powerhell script file: <$ HiveScript = "$ storageUri / Oozie / input / useooziewf.hql" # $ hiveScript = "$ warehouse / oz / input /" $ hiveTableName = "log4jlogs" $ hiveDataFolder = " $ StoreUri "$ HiveOutputFolder =" $ storageUri / OozieOutput "$ passwd = to convert- SecureString $ clusterPassword-AsPlainText-Force $ creds = New Object System. Management. Automation. PSCredential ($ cluster user's name, $ passwd) Usage- AzureHDInsightCluster $ cluster name $ OoziePayload = @ "& lt;? Xml version =" 1.0 "encoding =" UTF-8 "& gt; & lt; configuration & gt;

  • Property & gt; & lt; name & gt; nameNode & lt; / name & gt; & lt; value & gt; $ storageUri & lt; / value & gt; & lt; / property & gt; & lt; Property & gt; & lt; name & gt; Jo obTracker & lt; / name & gt; & lt; value & gt; jobtrackerhost: 9010 & lt; / pricing & gt; & gt; / property & gt; & lt; Property & gt; & lt; name & gt; queueName & lt; / name & gt; & lt; value & gt; default & lt; / pricing & gt; & gt; & gt; & gt; asset & gt; asset & gt; ; & Lt; name & gt; oozie.use.system.l ibpath & lt; / name & gt; & gt; value & gt; true & lt; / pricing & gt; & lt; / asset & gt; & lt; asset & gt; & lt; name & gt; hiveScript & lt ; / Name & gt; Price & gt; $ hiveScript & lt; / pricing & gt; & lt; / asset & gt; & lt; asset & gt; & lt; name & gt; hiveTableName & lt; / Name & gt; & lt; value & gt; $ hiveTableName & lt; / pricing & gt; & Lt; / Property & gt; & Lt; Property & gt; & Lt; Name & gt; HiveDataFolder & lt; / Name & gt; & Lt; Price & gt; $ HiveDataFolder & lt; / Pricing & gt; & Lt; / Property & gt; & Lt; Property & gt; & Lt; Name & gt; HiveOutputFolder & lt; / Name & gt; & Lt; Price & gt; $ HiveOutputFolder & lt; / Pricing & gt; & Lt; / Property & gt; & Lt; Property & gt; & Lt; Name & gt; User.name & lt; / Name & gt; & Lt; Price & gt; Admin & lt; / Pricing & gt; & Lt; / Property & gt; & Lt; Property & gt; & Lt; Name & gt; Oozie.wf.application.path & lt; / Name & gt; & Lt; Price & gt; $ OozieWFPath & lt; / Pricing & gt; & Lt; / Property & gt; & Lt; / Configuration & gt; "@

    To start the Ozzy job as below: 

      Create OZ Job Write-Host" sending the following payload in the cluster: "-Forgorounder Green - Host "` n -------- `$ $ OoziePayload_n --------" $ clusterUriCreateJob = "https: //$clusterName.zurehdinsight.net: 443 / oozie / v1 / jobs" $ response = Ahwan- RestMethod -Method post -Uri $ clusterUriCreateJob -Credential $ creds -Body $ OoziePayload -ContentType "application / xml" -OutVariable $ OozieJobName # -debug $ jsonResponse = ConvertFrom-Json (ConvertTo-Json -InputObject $ response) $ OozieJobId = $ jsonResponse [0]. ("ID") Type # - Host "Ozzy Job ID $ ooz IeJobId is ... "# Ozzy Job Writing Host" Ozzy Job is Starting $ oozieJobId ... "- ForegroundColor Green $ clusterUriStartJob ="? Action "on = enabled" https :: //$clusterName.azurehdinsight.net 443 / oozie / v1 / job / "+ $ oozieJobId + $ response = Ahwan- RestMethod -Method keep -Uri $ clusterUriStartJob - Credential $ Credits | format table -HideTableHeaders # -debug   
     hive jobs (HKL file e):  
      drop table $ {hiveTableName}; Create external table $ {hiveTableName} ( T1 string, t2 string, t3 string, T-4 string, T5 string, T6 string, T7 string) Line format delimited by the ground, 'textfile' The status '$ {hiveDataFolder}' is stored;    
     
     
     Assume that  $ oozieWFPath  An existing workflow xml, you can try adding parameters for the hive action:  
      & action; action name = "mehhyation">  hive xmlns = "uri: oozie: Hyp-action: 0.2" & gt; Work-tracker & gt; $ {jobTracker} & lt; / work-tracker & gt; & lt; name node & gt; $ {nameNode} & Lt; / Name node & gt; & Lt; Script & gt; $ {HiveScript} & lt; / Script & gt; & Lt; Param & gt; hiveTableName = $ {hiveTableName} & lt; / param & gt; & Lt; Param & gt; hiveDataFolder = $ {hiveDataFolder} & lt; / param & gt; & Lt; / Hive & gt; ... & lt; / Action>   
     Those last two ultimate nodes must pass the Oozie variable to the hive scripts.

    • Comments

    Post a Comment

    Popular posts from this blog

    Java - Error: no suitable method found for add(int, java.lang.String) -

    I'm in the middle of homework work and I'm stuck. At this point in my code, I think that I should have a GUI window that opens and allows me to type "inserted text number". Notice is not going anywhere but at this point, If I pass through the problem then it will be in a linked list. I am getting two of the same error for the lines. Add (index, element); And I can not seem to get past it, there is no suitable method for the error "add (int, java. string string)". Code is below, please advise. To clarify - this will not be a method error because it is a linked list. There should not be any way involved. import java.awt. *; Import java.awt.event. *; Import javax.swing. *; Import java.util. *; Import java.util.Scanner; Import java.util.LinkedList; Public class TopTenList JFrame {Private TopTenList tt; See the Private JTextArea list; Private JTextField cmdTextField; Private JTextField resultsTestfield; // This GUI window is the code for the public toptist...
    Read more

    java - JPA TypedQuery: Parameter value element did not match expected type -

    I am using JPA 2.0 and getting the following code in the DAO layer: Public Zero Test () {string key = "status"; String [] Conditions = {"A", "B"}; & Lt; TestTable & gt; Results = Search (Keys, Conditions); } Public listing & lt; TestTable & gt; Search (string key, object value error) {string sql = "test ndf to testlet and jade nand." + Key + "in:" + key; TypedQuery & LT; TestTable & gt; Query = entityManager.createQuery (SQL, TestTable.class); Query.setParameter (key, Arrays.asList (valueArr)) / *** Error Line *** Return query.getResultList (); } In the above error line, it throws the following exception: java.lang.IllegalArgumentException: parameter value element [[Ljava.lang.String; @ cbe5bc] did not match the expected type [java.lang.String] Why is this expected type string while actually this string []? Please help! Note: This is derived from the normal routine and is a simplified code. I can no...
    Read more

    c++ - static template member variable has internal linkage but is not defined -

    Yes, I know, there is a question with almost the same title, but it refers to a different situation. Error message) In my case, I have a .cpp file with a big named designation name (implementation details). There is a property class template with a static data member in that name space, which I need to access from outside the unknown namespace. I give it a little bit of meat: file.hpp namespace Bar {template & lt; Typename A & gt; Struct foo {static_assert (is_same & lt; a, float & gt; :: value} is_same & lt; a, double>: value, ""); Fixed zero set_static_var (a const & x); // ...}; } and file.cpp namespace {template & lt; Typename A & gt; Struct foo_traits {// supports the implementation of several static code bars: foo & lt; & Gt; Fixed A data; }; The template's & lt; & Gt; Float foo_traits & lt; Float & gt; :: datum; // No change if this is in the global namespace template & lt; & Gt; Doub...
    Read more