データ パイプライン

Considerations

API 経由でデータをエクスポートできることは、Jira のデータ パイプラインの完全自動化のための最初のマイルストーンです。このマイルストーンでは、既存のデータ管理プラットフォーム (Tableau や PowerBI など) とワークフローをすでに持っているチームに、データ抽出機能を提供することに重点を置いています。

このマイルストーンの特徴は次のとおりです。 

• データのエクスポートは Jira のパフォーマンスに影響するため、負荷の軽い時間中に実行することをおすすめします。
• Jira Core、Jira Software、Jira Service Management からのデータがエクスポートされます。これには、すべての主要な課題とそのフィールド、Jira Software 内の組み込みフィールド (スプリントやストーリー ポイントなど)、Jira Service Management (カスタマー リクエスト タイプや SLA など) が含まれます。
• 自分で作成したカスタム フィールドはエクスポートされません。

セキュリティ

エクスポートにはすべてのデータが含まれており、フィルターや変換に必要な情報をできるだけ多く得ることができます。この豊富なデータにより、チームによる Jira の使用状況から深い理解が柔軟に得られます。

同時にこのエクスポートでは、セキュリティと機密性に基づいてデータをフィルターしません。つまり、エクスポートするたびに、セキュリティで保護された機密データを自分で処理する必要があります。これに伴い、課題のスキーマには、これを実行するために使用できる security_level フィールドが含まれています。Jira の共有ホーム フォルダに十分なアクセス制限が設定されていることを確認してください。

パフォーマンスの影響

データのエクスポートは、アプリケーション ノード、データベース、および Lucene インデックスに影響を与える、リソースを大量に消費するプロセスです。内部テストでは、エクスポートをアクティブに実行しているノードのすべての製品機能について、5% のパフォーマンス低下が発生しました。次のことを強くおすすめします。

• データのエクスポートは、アクティビティが少ない時間帯またはアクティビティのないノードで実行します。
• fromDate パラメーターを使用してエクスポートするデータの量を制限します。日付が古過ぎると、より多くのデータがエクスポートされるため、データのエクスポート時間が長くなります。

また、テストの結果、課題数あたりのエクスポートのおおよその期間は以下のようになりました。

要件

この機能は、Jira Data Center ライセンスでのみ利用できます。

API を介して Jira の現在の状態データをエクスポートするには、API を介して Jira システム管理者としてログインする必要もあります。サポートされる API 認証方法の詳細については、「セキュリティの概要」を参照してください。

データ エクスポートの実行

Jira の現在の状態データをエクスポートするには、/jira/rest/datapipeline/latest/ ベース URL から /export REST API エンドポイントを使用します。

https://<host>:<port>/jira/rest/datapipeline/latest/export?fromDate=<yyyy-MM-ddTHH:mmTZD>

fromDate パラメーターは、エクスポートするデータの量を制限します。つまり、fromDate 値の後に作成または更新された課題に関するデータのみがエクスポートされます。fromDate パラメーターなしでエクスポートをトリガーすると、直近の 365 日内のすべてのデータがエクスポートされます。

/export REST API エンドポイントには、次の 3 つのメソッドがあります。

{
  "startTime": "2020-11-01T06-35-41-577+11",
  "nodeId": "mycluster2",
  "jobId": 124,
  "status": "STARTED"
}

{
  "startTime": "2020-11-01T06-35-41-577+11",
  "nodeId": "mycluster2",
  "jobId": 124,
  "status": "STARTED"
}

{
  "startTime": "2021-01-06T00:38:37.961Z",
  "completedTime": "2021-01-06T00:38:38.49Z",
  "nodeId": "mycluster2",
  "jobId": 124,
  "details": "Processed 0 entity(ies) and wrote 0 row(s) in total; Inconsistent index used for export job.",
  "status": "FAILED"
}

{}

{
  "startTime": "2020-11-01T06-35-41-577+11",
  "nodeId": "mycluster2",
  "jobId": 125,
  "status": "STARTED"
}

{
  "startTime": "2020-11-01T06-35-41-577+11",
  "completedTime": "2020-11-01T06-36-19-961+11",
  "nodeId": "mycluster2",
  "jobId": 125,
  "details": "Processed 23 entity(ies) and wrote 54 row(s) in total",
  "status": "COMPLETED"
}

{
  "startTime": "2020-11-02T04-20-34-007+11",
  "completedTime": "2020-11-02T04-22-10-961+11",
  "nodeId": "mycluster2",
  "jobId": 125,
  "status": "CANCELLATION_REQUESTED"
}

{
  "startTime": "2020-11-02T04-20-34-007+11",
  "cancelledTime": "2020-11-02T04-24-21-717+11",
  "completedTime": "2020-11-02T04-24-21-717+11",
  "nodeId": "mycluster2",
  "jobId": 125,
  "details": "Processed 23 entity(ies) and wrote 12 row(s) in total",
  "status": "CANCELLED"
}

{
  "status": "OK",
  "message": "Cancellation request successfully received. Currently running export job will be stopped shortly."
}

{
  "status": "WARNING",
  "message": "Cancellation request aborted. There is no export job running to cancel."
}

自動データ エクスポート キャンセル

データ エクスポートを実行するノードが正常にシャットダウンされると、Jira は自動的にエクスポートを CANCELLED としてマークします。

クラッシュまたはハードウェア レベルの障害が発生し、JVM に通知されなかった場合、エクスポート プロセスがロックされることがあります。この場合、DELETE リクエストを通じてエクスポートを CANCELLED としてマークする必要があります。これにより、プロセスがロックから解放され、別のデータ エクスポートを実行できます。

データ エクスポートを設定する

次のシステム プロパティを使用して、エクスポート データの形式を設定できます。

出力ファイル 

データ エクスポートを実行するたびに、Jira は数値ジョブ ID をタスクに割り当てます (最初のデータ エクスポートに対して 1 から開始)。このジョブ ID は、エクスポートされたデータを含むファイルのファイル名およびロケーションで使用されます。 

エクスポートされたデータは、/<jira-shared-home>/data-pipeline/export/<job_id>/: にある 3 つの CSV ファイルに保存されます。

• issues_job<job_id>_<timestamp>.csv (for issues)

  サンプル コンテンツを表示する

  id,instance_url,key,url,project_key,project_name,project_type,project_category,issue_type,summary,description,environment,creator_id,creator_name,reporter_id,reporter_name,assignee_id,assignee_name,status,status_category,priority_sequence,priority_name,resolution,watcher_count,vote_count,created_date,resolution_date,updated_date,due_date,estimate,original_estimate,time_spent,parent_id,security_level,labels,components,affected_versions,fix_versions
  10022,http://localhost:8090/jira,TP-23,http://localhost:8090/jira/browse/TP-23,TP,Test Project,Software,,Story,"As a user, I'd like a historical story to show in reports",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-10T08:44:19Z,2020-10-22T17:19:19Z,2020-10-22T17:19:19Z,,,,,,,,,,"[""Version 1.0""]"
  10021,http://localhost:8090/jira,TP-22,http://localhost:8090/jira/browse/TP-22,TP,Test Project,Software,,Story,"As a user, I'd like a historical story to show in reports",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-10T08:44:19Z,2020-10-20T15:41:19Z,2020-10-20T15:41:19Z,,,,,,,,,,"[""Version 1.0""]"
  10020,http://localhost:8090/jira,TP-21,http://localhost:8090/jira/browse/TP-21,TP,Test Project,Software,,Story,"As a user, I'd like a historical story to show in reports",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-10T08:44:19Z,2020-10-18T00:21:19Z,2020-10-18T00:21:19Z,,,,,,,,,,"[""Version 1.0""]"
  10019,http://localhost:8090/jira,TP-20,http://localhost:8090/jira/browse/TP-20,TP,Test Project,Software,,Story,"As a user, I'd like a historical story to show in reports",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-10T08:44:19Z,2020-10-15T17:43:19Z,2020-10-15T17:43:19Z,,,,,,,,,,"[""Version 1.0""]"
  10018,http://localhost:8090/jira,TP-19,http://localhost:8090/jira/browse/TP-19,TP,Test Project,Software,,Story,"As a user, I'd like a historical story to show in reports",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-10T08:44:19Z,2020-10-14T05:08:19Z,2020-10-14T05:08:19Z,,,,,,,,,,"[""Version 2.0""]"
  10017,http://localhost:8090/jira,TP-18,http://localhost:8090/jira/browse/TP-18,TP,Test Project,Software,,Story,"As a user, I'd like a historical story to show in reports",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-10T08:44:19Z,2020-10-11T05:14:19Z,2020-10-11T05:14:19Z,,,,,,,,,,"[""Version 2.0""]"
  10016,http://localhost:8090/jira,TP-17,http://localhost:8090/jira/browse/TP-17,TP,Test Project,Software,,Bug,"Instructions for deleting this sample board and project are in the description for this issue >> Click the ""TP-17"" link and read the description tab of the detail view for more","*To delete this Sample Project _(must be performed by a user with Administration rights)_* \n- Open the administration interface to the projects page by using the keyboard shortcut 'g' then 'g' and typing 'Projects' in to the search dialog\n- Select the ""Delete"" link for the ""Test Project"" project\n\n*To delete the Sample Project workflow and workflow scheme _(must be performed by a user with Administration rights)_* \n- Open the administration interface to the workflow schemes page by using the keyboard shortcut 'g' then 'g' and typing 'Workflow Schemes' in to the search dialog\n- Select the ""Delete"" link for the ""TP: Software Simplified Workflow Scheme"" workflow scheme\n- Go to the workflows page by using the keyboard shortcut 'g' then 'g' and typing 'Workflows' in to the search dialog(_OnDemand users should select the second match for Workflows_)\n- Expand the ""Inactive"" section\n- Select the ""Delete"" link for the ""Software Simplified Workflow  for Project TP"" workflow\n\n*To delete this Board _(must be performed by the owner of this Board or an Administrator)_*\n- Click the ""Tools"" cog at the top right of this board\n- Select ""Delete""",,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,Done,Done,3,Medium,Done,0,0,2020-10-24T09:54:19Z,2020-10-28T02:30:19Z,2020-10-28T02:30:19Z,,,,,,,,,,
  10015,http://localhost:8090/jira,TP-16,http://localhost:8090/jira/browse/TP-16,TP,Test Project,Software,,Story,"As a team, we can finish the sprint by clicking the cog icon next to the sprint name above the ""To Do"" column then selecting ""Complete Sprint"" >> Try closing this sprint now",,,10000,Marek Szczepański,10000,Marek Szczepański,,,Done,Done,3,Medium,Done,0,0,2020-10-03T13:50:19Z,2020-10-25T16:26:19Z,2020-10-25T16:26:19Z,,,,,,,,,,
  10014,http://localhost:8090/jira,TP-15,http://localhost:8090/jira/browse/TP-15,TP,Test Project,Software,,Story,"As a scrum master, I can see the progress of a sprint via the Burndown Chart >> Click ""Reports"" to view the Burndown Chart",,,10000,Marek Szczepański,10000,Marek Szczepański,,,Done,Done,3,Medium,Done,0,0,2020-10-24T09:54:19Z,2020-10-29T00:30:19Z,2020-10-29T00:30:19Z,,,,,,,,,,
  10013,http://localhost:8090/jira,TP-14,http://localhost:8090/jira/browse/TP-14,TP,Test Project,Software,,Story,"As a user, I can find important items on the board by using the customisable ""Quick Filters"" above >> Try clicking the ""Only My Issues"" Quick Filter above",*Creating Quick Filters*\n\nYou can add your own Quick Filters in the board configuration (select *Board > Configure*),,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,To Do,To Do,3,Medium,,0,0,2020-10-30T14:54:19Z,,2020-10-30T14:54:19Z,,,,,,,,,,
  10012,http://localhost:8090/jira,TP-13,http://localhost:8090/jira/browse/TP-13,TP,Test Project,Software,,Bug,"As a developer, I can update details on an item using the Detail View >> Click the ""TP-13"" link at the top of this card to open the detail view","*Editing using the Detail View*\n\nMany of the fields in the detail view can be inline edited by simply clicking on them. \n\nFor other fields you can open Quick Edit, select ""Edit"" from the cog drop-down.",,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,To Do,To Do,3,Medium,,0,0,2020-10-24T09:54:19Z,,2020-10-24T09:54:19Z,,,,,,,,,,"[""Version 2.0""]"
  10011,http://localhost:8090/jira,TP-12,http://localhost:8090/jira/browse/TP-12,TP,Test Project,Software,,Sub-task,"When the last task is done, the story can be automatically closed >> Drag this task to ""Done"" too",,,10000,Marek Szczepański,10000,Marek Szczepański,,,In Progress,In Progress,3,Medium,,0,0,2020-10-28T19:21:19Z,,2020-10-28T19:21:19Z,,,,,10009,,,,,"[""Version 2.0""]"
  10010,http://localhost:8090/jira,TP-11,http://localhost:8090/jira/browse/TP-11,TP,Test Project,Software,,Sub-task,"Update task status by dragging and dropping from column to column >> Try dragging this task to ""Done""",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,In Progress,In Progress,3,Medium,,0,0,2020-10-27T21:56:19Z,,2020-10-27T21:56:19Z,,,,,10009,,,,,"[""Version 2.0""]"
  10009,http://localhost:8090/jira,TP-10,http://localhost:8090/jira/browse/TP-10,TP,Test Project,Software,,Story,"As a developer, I can update story and task status with drag and drop (click the triangle at far left of this story to show sub-tasks)",,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,In Progress,In Progress,3,Medium,,0,0,2020-10-24T09:54:19Z,,2020-10-24T09:54:19Z,,,,,,,,,,"[""Version 2.0""]"
  10008,http://localhost:8090/jira,TP-9,http://localhost:8090/jira/browse/TP-9,TP,Test Project,Software,,Story,"As a developer, I'd like to update story status during the sprint >> Click the Active sprints link at the top right of the screen to go to the Active sprints where the current Sprint's items can be updated",,,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:23Z,,2020-10-31T19:04:23Z,,,,,,,,,,
  10007,http://localhost:8090/jira,TP-8,http://localhost:8090/jira/browse/TP-8,TP,Test Project,Software,,Bug,"As a product owner, I'd like to include bugs, tasks and other issue types in my backlog >> Bugs like this one will also appear in your backlog but they are not normally estimated","*Estimation of Bugs*\n\nScrum teams do not normally apply story point estimates to bugs because bugs are considered to be part of the ongoing work that the team must deal with (i.e the overhead). If you view the story points completed in a sprint as a measure of progress, then bugs also have no value because they do not deliver anything additional to the customer. \n\nHowever, you can apply estimates to bugs if you wish by configuring the ""Story Points"" field to apply to other Issue Types (by default it only applies to Stories and Epics). Some more information on this is in the [documentation|https://confluence.atlassian.com/display/GH/Story+Point].",,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:23Z,,2020-10-31T19:04:23Z,,,,,,,,,,"[""Version 2.0""]"
  10006,http://localhost:8090/jira,TP-7,http://localhost:8090/jira/browse/TP-7,TP,Test Project,Software,,Sub-task,This is a sample task. Tasks are used to break down the steps to implement a user story,,,10000,Marek Szczepański,10000,Marek Szczepański,10000,Marek Szczepański,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:23Z,,2020-10-31T19:04:23Z,,,,,10005,,,,,
  10005,http://localhost:8090/jira,TP-6,http://localhost:8090/jira/browse/TP-6,TP,Test Project,Software,,Story,"As a scrum master, I'd like to break stories down into tasks we can track during the sprint >> Try creating a task by clicking the Sub-Tasks tab in the Detail View on the right",*Task Breakdown*\n\nMany teams choose to break down user stories into a set of tasks needed to implement the story. They then update the status of these tasks during a sprint to track progress. The completion of the last task signals the end of the story. \n\nYou can add sub-tasks to a story on the sub-task tab (folder icon) above.,,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:22Z,,2020-10-31T19:04:22Z,,,,,,,,,,
  10004,http://localhost:8090/jira,TP-5,http://localhost:8090/jira/browse/TP-5,TP,Test Project,Software,,Story,"As a team, I'd like to commit to a set of stories to be completed in a sprint (or iteration) >> Click ""Create Sprint"" then drag the footer down to select issues for a sprint (you can't start a sprint at the moment because one is already active)","*Starting a Sprint*\n\nDuring the Planning Meeting the team will examine the stories at the top of the backlog and determine which they can commit to completing during the coming sprint. Based on this information the Product Owner might break down stories into smaller stories, adjust story priorities or otherwise work with the team to define the ideal sprint outcome. When the sprint is started the stories are moved into the sprint backlog.",,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:21Z,,2020-10-31T19:04:21Z,,,,,,,,,,
  10003,http://localhost:8090/jira,TP-4,http://localhost:8090/jira/browse/TP-4,TP,Test Project,Software,,Story,"As a team, I'd like to estimate the effort of a story in Story Points so we can understand the work remaining >> Try setting the Story Points for this story in the ""Estimate"" field","This story is estimated at 5 Story Points (as shown in the ""Estimate"" field at the top right of the Detail View). \n\nTry updating the Story Point estimate to 4 by clicking on the ""Estimate"" then typing.\n\n*Estimating using Story Points*\n\nBecause the traditional process of estimating tasks in weeks or days is often wildly inaccurate, many Scrum teams estimate in Story Points instead. Story Points exist merely as a way to estimate a task's difficulty compared to some other task (for example, a 10-point story would probably take double the effort of a 5-point story). As teams mature with Scrum they tend to achieve a consistent number of Story Points from Sprint to Sprint -- this is termed the team's _velocity_. This allows the Product Owner to use the velocity to predict how many Sprints it will take to deliver parts of the backlog. \n\nMany teams use Planning Poker to achieve consensus on Story Point estimates.\n\n*Using Other Estimation Units*\n\nYou can configure JIRA Software to use time-based estimates if you wish. In the configuration for the board, on the ""Estimation"" tab, select ""Original Time Estimate"" as your Estimation Statistic. If you also wish to track the time used during the Sprint, select ""Remaining Estimate and Time Spent"" to enable Time Tracking in JIRA Software.",,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:21Z,,2020-10-31T19:04:21Z,,,,,,,,,,"[""Version 3.0""]"
  10002,http://localhost:8090/jira,TP-3,http://localhost:8090/jira/browse/TP-3,TP,Test Project,Software,,Story,"As a product owner, I'd like to rank stories in the backlog so I can communicate the proposed implementation order >> Try dragging this story up above the previous story",*About the Product Backlog*\n\nThe backlog is the source of truth for the order of work to be completed. It is expected that the Product Owner will work with the team to make sure that the backlog represents the current approach to delivering the product. JIRA Software makes it easy to prioritise (rank) Stories by dragging them up and down the backlog.,,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:21Z,,2020-10-31T19:04:21Z,,,,,,,,,,"[""Version 3.0""]"
  10001,http://localhost:8090/jira,TP-2,http://localhost:8090/jira/browse/TP-2,TP,Test Project,Software,,Story,"As a product owner, I'd like to express work in terms of actual user problems, aka User Stories, and place them in the backlog >> Try creating a new story with the ""+ Create Issue"" button (top right of screen)","When you click ""+ Create Issue"" you will be asked for the correct project (select ""Test Project"") and Issue Type (select ""Story"").\n\n*About User Stories*\n\nThe Scrum methodology drops traditional software requirement statements in favour of real world problems expressed as User Stories. Stories describe the task a particular user is trying to achieve and its value. They are typically of the form ""As a (role) I want (something) so that (benefit)"". This approach focuses the team on the core user need rather than on implementation details. \n\nStories are ""placeholders for a conversation"" -- they do not need to be especially detailed since it is expected that the team will work together to resolve ambiguity as the story is developed. \n\nStories to be implemented in the future are stored in the Product Backlog. The backlog is ranked by the Product Owner so that the next items to be completed are at the top.",,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:21Z,,2020-10-31T19:04:21Z,,,,,,,,,,"[""Version 2.0""]"
  10000,http://localhost:8090/jira,TP-1,http://localhost:8090/jira/browse/TP-1,TP,Test Project,Software,,Story,"As an Agile team, I'd like to learn about Scrum >> Click the ""TP-1"" link at the left of this row to see detail in the Description tab on the right","*About Scrum*\n\nScrum is an iterative approach to Agile software development. The methodology has been around since the 1980s but was popularised by Jeff Sutherland and Ken Schwaber. \n\nScrum breaks the development of a product down in to discrete iterations (termed Sprints) that each deliver functionality that could potentially be shipped to users.\n\nThe Scrum Alliance offers an excellent [introduction to Scrum|http://www.scrumalliance.org/resources/47] that provides an overview of key Scrum concepts, stakeholders, processes and artefacts.",,10000,Marek Szczepański,10000,Marek Szczepański,,,To Do,To Do,3,Medium,,0,0,2020-10-31T19:04:19Z,,2020-10-31T19:04:19Z,,,,,,,,,,"[""Version 2.0""]"

• issue_fields_job<job_id>_<timestamp>.csv (Jira Software フィールドと Jira Service Management フィールドの場合)

  サンプル コンテンツを表示する

  issue_id,field_id,field_name,field_value
  10022,story_points,Story Points,2.0
  10022,sprint,Sprint,"[{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10021,story_points,Story Points,2.0
  10021,sprint,Sprint,"[{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10020,story_points,Story Points,1.0
  10020,sprint,Sprint,"[{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10019,story_points,Story Points,3.0
  10019,sprint,Sprint,"[{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10018,story_points,Story Points,5.0
  10018,sprint,Sprint,"[{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10017,story_points,Story Points,3.0
  10017,sprint,Sprint,"[{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10016,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10015,story_points,Story Points,2.0
  10015,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null},{""id"":2,""name"":""Sample Sprint 1"",""goal"":null,""boardId"":1,""state"":""CLOSED"",""startDate"":""2020-10-10T08:44:26Z"",""endDate"":""2020-10-24T08:44:26Z"",""completeDate"":""2020-10-24T07:24:26Z""}]"
  10014,story_points,Story Points,4.0
  10014,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10013,story_points,Story Points,3.0
  10013,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10012,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10011,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10010,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10009,story_points,Story Points,5.0
  10009,sprint,Sprint,"[{""id"":1,""name"":""Sample Sprint 2"",""goal"":null,""boardId"":1,""state"":""ACTIVE"",""startDate"":""2020-10-24T09:54:23Z"",""endDate"":""2020-11-07T10:14:23Z"",""completeDate"":null}]"
  10008,story_points,Story Points,3.0
  10005,story_points,Story Points,1.0
  10004,story_points,Story Points,1.0
  10003,story_points,Story Points,5.0
  10002,story_points,Story Points,5.0
  10001,story_points,Story Points,2.0
  10000,story_points,Story Points,2.0

• sla_cycles_job<job_id>_<timestamp>.csv (Jira Service Management がインストールされている場合の SLA サイクル情報の場合)
  

  サンプル コンテンツを表示する

  issue_id, sla_id, sla_name, cycle_type, start_time, stop_time, paused, goal_duration, elapsed_time, remaining_time
  
  10000, 1, Time to first response, Ongoing, 2020-01-10T12:50:30Z, 2020-01-10T12:50:30Z, true, 14400000, 14400000, 14400000
  10000, 1, Time to first response, Completed, 2020-01-10T12:50:30Z, 2020-01-10T12:50:30Z,, 14400000, 14400000, 14400000
  10000, 1, Time to first response, Completed, 2020-01-10T12:50:30Z, 2020-01-10T12:50:30Z,, 14400000, 14400000, 14400000
  10000, 1, Time to first response, Completed, 2020-01-10T12:50:30Z, 2020-01-10T12:50:30Z,, 14400000, 14400000, 14400000
  10000, 2, Time to approve normal change, Completed, 2020-01-10T12:50:30Z, 2020-01-10T12:50:30Z,, 14400000, 14400000, 14400000
  10000, 2, Time to approve normal change, Completed, 2020-01-10T12:50:30Z, 2020-01-10T12:50:30Z,, 14400000, 14400000, 14400000

このエクスポートでデータをロードおよび変換するには、そのスキーマを理解する必要があります。スキーマの詳細については、「データ パイプライン エクスポート スキーマ」を参照してください。

Spark と Hadoop のインポート設定の例

既存の Spark インスタンスまたは Hadoop インスタンスがある場合、次の参照を使用し、さらに変換するためにデータをインポートする方法を設定します。

%python
# File location. Change to s3 once we have our own zone
file_location = "/FileStore/**/export_2020_09_24T03_32_18Z.csv" 

# Automatically set data type for columns
infer_schema = "true"
# Skip first row as it's a header
first_row_is_header = "true"
# Ignore multiline within double quotes
multiline_support = "true"

# The applied options are for CSV files. For other file types, these will be ignored. Note escape & quote options for RFC-4801 compliant files
df = spark.read.format("csv") \
  .option("inferSchema", infer_schema) \
  .option("header", first_row_is_header) \
  .option("multiLine", multiline_support) \
  .option("quote", "\"") \
  .option("escape", "\"") \
  .option("encoding", "UTF-8").load(file_location)

display(df)

CREATE EXTERNAL TABLE IF NOT EXISTS some_db.datapipeline_export (
  `id` string,
  `instance_url` string,
  `key` string,
  `url` string,
  `project_key` string,
  `project_name` string,
  `project_type` string,
  `project_category` string,
  `issue_type` string,
  `summary` string,
  `description` string,
  `environment` string,
  `creator_id` string,
  `creator_name` string,
  `reporter_id` string,
  `reporter_name` string,
  `assignee_id` string,
  `assignee_name` string,
  `status` string,
  `status_category` string,
  `priority_sequence` string,
  `priority_name` string,
  `resolution` string,
  `watcher_count` string,
  `vote_count` string,
  `created_date` string,
  `resolution_date` string,
  `updated_date` string,
  `due_date` string,
  `estimate` string,
  `original_estimate` string,
  `time_spent` string,
  `parent_id` string,
  `security_level` string,
  `labels` string,
  `components` string,
  `affected_versions` string,
  `fix_versions` string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
  "escapeChar" = "\\",
  'quoteChar' = '"',
  'separatorChar' = ','
) LOCATION 's3://bulldog-data-pipeline/test-exports/'
TBLPROPERTIES ('has_encrypted_data'='false');