Skip to content

Recreating server-regression QuickSight Dashboards

Follow this runbook in the event that the QuickSight dashboards for the server-regression's test suite's performance metrics are ever destroyed and need to be recreated.

As of writing, the following dashboards are available to view for members of the BFD team:

Note: The above dashboards can also be found in QuickSight's "Dashboards" view

Glossary

Term Definition
server-regression Technical identifier for the Locust-based performance regression test suite that runs against BFD Server during deployment in all environments
QuickSight An AWS Service that allows for the creation of visual dashboards from various datasets (i.e. S3, RDS, Athena, etc.)
Dataset QuickSight's term for a data store (such as S3, Athena, etc) along with the fields to display from the store
Analysis A work-in-progress Dashboard that uses a Dataset to display fields
Dashboard A collection of graphs and visuals that represents a Dataset
Athena An AWS Service that is used to analyze data stored in AWS S3 using standard SQL

FAQ

Will the hash in each of the Dataset SQL queries below ever change? What is it used for?

Firstly, the hash (or, more specifically, metadata.hash) is a SHA256 hash of various test parameters and metadata values from a given test suite run. Performance metrics with the same metadata.hash value can be confidently compared as they were collected from test suite runs that were started with the same parameters (i.e. number of simulated users, desired test runtime, types of tests run, etc.).

There are really only two scenarios where the hash could change:

  1. The parameters (runtime, user count, spawn rate) that control how the server-regression test suite runs during BFD Server deployments change from their current defaults
  2. The code to generate the hash changes because we add an additional field or want to change the hashing algorithm

These scenarios are unlikely for the following reasons:

  1. With the current parameters, consistent and useful results are being captured with minimal increase in deployment time. That is to say, at time of writing there is no benefit in changing these parameters
  2. There is no additional metadata that Locust exposes that we would want to include in the hash to further delineate performance metrics that can or can't be compared

If, however, the hash does change in the future, the queries listed below will need to be updated in both this runbook and in each of the Datasets in QuickSight. Note that this scenario will be easy to identify, as the visuals in each Dashboard will stop displaying data.

Instructions

  1. You will need to navigate to the QuickSight AWS Service in order to start creating QuickSight Dashboards

  2. Navigate to aws.amazon.com in any browser

  3. Click the "Sign In" link in the navigation bar on the top right of the screen
  4. Enter your AWS account credentials and sign-in
  5. You should now have navigated to the "Console Home", click the search input in the navigation bar on the top left of the screen

  6. Type "QuickSight" and click the QuickSight result that appears

  7. Every QuickSight dashboard requires a Dataset, which for the server-regression dashboards will be a Custom SQL query against Athena. As there will be three dashboards, there will be three datasets (one dataset for each dashboard)

  8. Creating the bfd-test-server-regression dataset:

    1. Navigate to "Datasets" in QuickSight (click on "Datasets" in the navigation tree on the left side of the page)
    2. Click the "New dataset" button on the top-left of the page
    3. The page that appears should list a large number of possible data sources. Find "Athena" in this list, and click on it
    4. Enter "bfd-test-server-regression" for the "Data source name"
    5. Choose bfd for the "Athena workgroup"
    6. Click "Create data source"
    7. Choose AwsDataCatalog as the "Catalog"
    8. Choose bfd-insights-bfd-test as the "Database"
    9. Choose bfd_insights_bfd_test_server_regression as the "Table"
    10. Click "Use custom SQL"
    11. In the first input box, enter "bfd-test-server-regression" (replacing "New custom SQL")
    12. In the large text area, enter the following SQL query:
    select
  totals.total_reqs_per_second,
  totals.num_requests as total_requests,
  totals.num_failures as total_failures,
  totals.min_response_time as total_min_response_time,
  totals.average_response_time as total_avg_response_time,
  totals.median_response_time as total_median_response_time,
  totals.response_time_percentiles."0.95" as total_95_percentile,
  totals.response_time_percentiles."0.99" as total_99_percentile,
  totals.response_time_percentiles."1.0" as total_100_percentile,
  metadata.compare_result,
  date_format(from_unixtime(metadata.timestamp, 'US/Eastern'), '%b-%e-%y %T EST') as date_time,
  metadata.timestamp as raw_timestamp
from "bfd-insights-bfd-test"."bfd_insights_bfd_test_server_regression"
where
  metadata.hash = '85d93b342368c8f57f8e78ea6c56979ad5e32901776aaeaffdd8467e4cee1df9'
  and contains(metadata.tags, 'master')
  and (
     metadata.validation_result = 'PASSED'
     or metadata.validation_result = 'NOT_APPLICABLE'
  )
order by metadata.timestamp asc
    1. Click "Confirm query"
    2. In "Finish dataset creation", select "Directly query your data"
    3. Click "Edit/preview data"
    4. An editor should appear with the SQL you entered above, click "Apply"
    5. Ensure that data is returned in the data preview
    6. Click "SAVE & PUBLISH" in the navigation bar on the top right of the screen
    7. Ensure that an in-page notification appears indicating the dataset was saved successfully
    8. Click "CANCEL" in the navigation bar on the top right of the screen to exit the Dataset creation page and return to QuickSight Home

  9. Creating the bfd-prod-sbx-server-regression dataset:

    1. Navigate to "Datasets" in QuickSight (click on "Datasets" in the navigation tree on the left side of the page)
    2. Click the "New dataset" button on the top-left of the page
    3. The page that appears should list a large number of possible data sources. Find "Athena" in this list, and click on it
    4. Enter "bfd-prod-sbx-server-regression" for the "Data source name"
    5. Choose bfd for the "Athena workgroup"
    6. Click "Create data source"
    7. Choose AwsDataCatalog as the "Catalog"
    8. Choose bfd-insights-bfd-prod-sbx as the "Database"
    9. Choose bfd_insights_bfd_prod_sbx_server_regression as the "Table"
    10. Click "Use custom SQL"
    11. In the first input box, enter "bfd-prod-sbx-server-regression" (replacing "New custom SQL")
    12. In the large text area, enter the following SQL query:
    select
  totals.total_reqs_per_second,
  totals.num_requests as total_requests,
  totals.num_failures as total_failures,
  totals.min_response_time as total_min_response_time,
  totals.average_response_time as total_avg_response_time,
  totals.median_response_time as total_median_response_time,
  totals.response_time_percentiles."0.95" as total_95_percentile,
  totals.response_time_percentiles."0.99" as total_99_percentile,
  totals.response_time_percentiles."1.0" as total_100_percentile,
  metadata.compare_result,
  date_format(from_unixtime(metadata.timestamp, 'US/Eastern'), '%b-%e-%y %T EST') as date_time,
  metadata.timestamp as raw_timestamp
from "bfd-insights-bfd-prod-sbx"."bfd_insights_bfd_prod_sbx_server_regression"
where
  metadata.hash = '71a5310d52e0ffb8c03dc74cdee7bf635bc7fbd75dfe1d8f647fcfb11e5b65ce'
  and contains(metadata.tags, 'master')
  and (
     metadata.validation_result = 'PASSED'
     or metadata.validation_result = 'NOT_APPLICABLE'
  )
order by metadata.timestamp asc
    1. Click "Confirm query"
    2. In "Finish dataset creation", select "Directly query your data"
    3. Click "Edit/preview data"
    4. An editor should appear with the SQL you entered above, click "Apply"
    5. Ensure that data is returned in the data preview
    6. Click "SAVE & PUBLISH" in the navigation bar on the top right of the screen
    7. Ensure that an in-page notification appears indicating the dataset was saved successfully
    8. Click "CANCEL" in the navigation bar on the top right of the screen to exit the Dataset creation page and return to QuickSight Home

  10. Creating the bfd-prod-server-regression dataset:

    1. Navigate to "Datasets" in QuickSight (click on "Datasets" in the navigation tree on the left side of the page)
    2. Click the "New dataset" button on the top-left of the page
    3. The page that appears should list a large number of possible data sources. Find "Athena" in this list, and click on it
    4. Enter "bfd-prod-server-regression" for the "Data source name"
    5. Choose bfd for the "Athena workgroup"
    6. Click "Create data source"
    7. Choose AwsDataCatalog as the "Catalog"
    8. Choose bfd-insights-bfd-prod as the "Database"
    9. Choose bfd_insights_bfd_prod_server_regression as the "Table"
    10. Click "Use custom SQL"
    11. In the first input box, enter "bfd-prod-server-regression" (replacing "New custom SQL")
    12. In the large text area, enter the following SQL query:
    select
  totals.total_reqs_per_second,
  totals.num_requests as total_requests,
  totals.num_failures as total_failures,
  totals.min_response_time as total_min_response_time,
  totals.average_response_time as total_avg_response_time,
  totals.median_response_time as total_median_response_time,
  totals.response_time_percentiles."0.95" as total_95_percentile,
  totals.response_time_percentiles."0.99" as total_99_percentile,
  totals.response_time_percentiles."1.0" as total_100_percentile,
  metadata.compare_result,
  date_format(from_unixtime(metadata.timestamp, 'US/Eastern'), '%b-%e-%y %T EST') as date_time,
  metadata.timestamp as raw_timestamp
from "bfd-insights-bfd-prod"."bfd_insights_bfd_prod_server_regression"
where
  metadata.hash = '727a44e865e717728ac409e8e23a7e13ce73a155874f3832a0caef1fde17e95c'
  and contains(metadata.tags, 'master')
  and (
     metadata.validation_result = 'PASSED'
     or metadata.validation_result = 'NOT_APPLICABLE'
  )
order by metadata.timestamp asc
    1. Click "Confirm query"
    2. In "Finish dataset creation", select "Directly query your data"
    3. Click "Edit/preview data"
    4. An editor should appear with the SQL you entered above, click "Apply"
    5. Ensure that data is returned in the data preview
    6. Click "SAVE & PUBLISH" in the navigation bar on the top right of the screen
    7. Ensure that an in-page notification appears indicating the dataset was saved successfully
    8. Click "CANCEL" in the navigation bar on the top right of the screen to exit the Dataset creation page and return to QuickSight Home

  11. Navigate to "Analyses" in QuickSight (click on "Analyses" in the navigation tree on the left side of the page from the QuickSight "home")

  12. Click "New analysis" on the top right of the page
  13. Choose bfd-test-server-regression
  14. A new page should appear with details about the dataset. Click on "USE IN ANALYSIS"
  15. A new page should appear with an empty "AutoGraph" and a variety of controls on the left-side:
  16. New Analysis Startpage

  17. Start by re-creating all of the line graphs for the server-regression dashboard:

  18. Click anywhere within the empty "AutoGraph". An outline should appear indicating it has been selected, along with controls in the top-right corner

  19. In "Visual types", choose a line graph (red outlined button in figure):
    1. Line Graph in Visual Types
  20. The graph's title/description should change to "You need to add or remove fields" and the collapsed "Field wells" section should expand showing "X axis", "Value", and "Color" as shown in the following figure:
    1. Line Graph Selected
  21. In the "Fields list", click and drag "date_time" to the "X axis" control in "Field wells". You should see the graph update after doing so
  22. In the "Fields list", click and drag "total_reqs_per_second" to the "Value" control in "Field wells". You should see the graph update, and the page should look like the following:
    1. Fields Added to Line Graph Without Change
  23. The graph's "Value" is currently representing the unique count of the "total_reqs_per_second" which is not correct. Click on "total_reqs_per_second (Count)" in the "Value" control in the "Field wells" section. A dropdown menu should appear
  24. Click on "Aggregate: Count". An additional dropdown menu should appear to the right of the menu
  25. Click on "Sum" in the list of aggregations
  26. The graph should update and the values being graphed should be correct, as shown below (note the values will be different, but the graph should now be graphing the actual value of total_reqs_per_second):
    1. "total_reqs_per_second" Changed to Sum Aggregation
  27. The graph's x-axis (date_time) is currently sorted alphabetically and not by time. Similar to above, click on "date_time" in the "X axis" control in "Field wells". A dropdown menu should appear. The first item in the menu should be "Sort by" and should be "total_reqs_per_second" as shown in the figure:
    1. "date_time" Dropdown "Sort by"
  28. Click on the "Sort by" option (highlighted by the red outline in the figure above). A second dropdown should appear to the right of the original menu
  29. Click on "Sort options" in the second, right-most dropdown
  30. The left pane (where "Fields list" was) should be replaced by "Sort options". In the select/dropdown menu labeled "Sort by", select "raw_timestamp" as shown in the figure below:
    1. "Sort by" Set to "raw_timestamp"
  31. Click "Apply" at the bottom of the "Sort options" pane. The graph should update and now be ordered correctly by time
  32. Click "Close". The left-most pane should switch back to "Fields list"
  33. Change the title of the graph to "Total RPS Over Time" by double-clicking on the current title ("Sum of Total_reqs_per_second by Date_time") and entering the new title in the text area
  34. Double-click the y-axis label ("total_reqs_per_second (Sum)", shown vertically). A dropdown menu should appear
  35. Select "Rename" from the dropdown menu
  36. Rename to "Total Requests-Per-Second"
  37. Follow the same steps as above to rename the x-axis, but this time the name should be "Date and Time (EST)"
  38. Hover over the graph and click the small pencil icon in the top right corner. Hovering over the icon should show a tooltip with the text "Format visual". The "Fields list" pane should be replaced by a pane named "Format visual" on the leftside of the page, as shown below:
    1. "Format visual" Pane
  39. There should be multiple collapsible sections, click on "Data labels"
  40. Click on the "Show data labels" checkbox. Each point on the line graph should now be labeled with its value
  41. Hover over the graph again and this time click the three-dots in the top right corner. A dropdown menu should appear
  42. Click "Duplicate visual"
  43. A duplicate of the original "Total RPS Over Time" line graph should appear to the right
  44. Click on this new, duplicate graph. It should become outlined
  45. Uncollapse the "Field wells" section above the graphs by clicking anywhere within it
  46. Click on the green "total_reqs_per_second (Sum)" button in the "Value" section. A dropdown menu should appear
  47. In the very bottom of the dropdown menu, beneath the "Search fields" input, find "total_requests". Click on it
  48. The graph's y-axis should change to "total_requests (Sum)" and the "Value" of the graph should now be "total_requests (Sum)"
  49. Change the title of the graph to "Total Requests over Time" following instructions above
  50. Change the title of the y-axis to "Total Requests" following instructions above
  51. Ensure the title of the x-axis remains "Date and Time (EST)". If not, change it following instructions above
  52. Ensure that data labels appear above each point in the graph. If not, change it following instructions above
  53. Duplicate any of the two graphs following the instructions above. A third graph should appear on bottom-left of the screen
  54. Following instructions outlined above, change the "Value"/y-axis of the graph to "total_failures"
  55. Change the title to "Number of Total Request Failures over Time"
  56. Change the y-axis title to "Total Failures"
  57. Ensure the x-axis title is "Date and Time (EST)"
  58. Ensure data labels appear above each point in the line graph
  59. Navigate to "Format visual" (ensure that the new failures graph is selected)
  60. In the "Title" collapsible section, click the "Edit subtitle" button

  61. Change the subtitle to "Successful runs should always have 0 failures"

  62. Next, recreate the totals bar charts

  63. On the top-left of the page, click the "ADD" button. A dropdown menu should appear

  64. Select "Add visual". A new "AutoGraph" should appear on the page
  65. From "Visual types" (bottom collapsible section of the left-pane) select "Vertical bar chart" (top-most row, right-most icon). Vertical bar charts require both an x-axis and values, but since these charts will be used to display values without any meaningful x-axis to plot them against we will need to create a fake x-axis value to plot against
  66. Click the "ADD" button again. Choose "Add calculated field". A new page should open with three main sections: a name field with the value "Add name", a section on the right with inner collapsible sections showing "Fields", "Parameters" and "Functions", and a large text area that takes up most of the screen
  67. In the large text area, type "1"
  68. For the name of the field, enter "fake_x". See the figure below:
    1. "fake_x" Calculated Field
  69. Click "Save". You should now return to the main Analysis screen with the graphs
  70. Select the empty vertical bar chart graph
  71. In "Field wells", set the x-axis to "fake_x"
  72. Set the "Value" to "total_avg_response_time". When set, the aggregate function should default to "Count"
  73. Following instructions outlined above to change the aggregate for a value, change the aggregate function from "Count" to "Min"
  74. Add another copy/duplicate of "totalavg_response_time" to the graph's "Value" (by dragging "total_avg_response_time" to "Value" in "Field wells" again) _beneath the previous "total_avg_response_time (Min)" value, as shown in the figure below:
  75. Multiple "total_avg_response_time" Values for Bar Chart
  76. This time, change the aggregate function from "Count" to "Median"
  77. Repeat the above steps two more times: add two more "total_avg_reponse_time" values, set the first's aggregate to "Average", and set the second's to "Max". When finished, "Values" should have 4 "total_avg_response_time" values each with a different aggregate function. See figure below:
  78. All "total_avg_respone_time" Values for Bar Chart
  79. Go to "Format visual" for the bar chart
  80. Un-collapse "X-axis" in "Format visual" by clicking on it
  81. Un-select every checkbox in the section: "Show title", "Show sort", "Show data zoom", "Show axis line", and "Show labels". The bar chart should now only display the y-axis
  82. Un-collapse "Legend"
  83. Un-select every checkbox in the section: "Show legend" and "Show legend title". The legend to the right of the bars in the vertical bar chart should no longer be visible
  84. Un-collapse "Data labels"
  85. Select "Show data labels". Each bar's value should now appear above their respective bar
  86. Un-collapse "Tooltip"
  87. Under "Display options", un-select "Use primary value as title" and select "Show aggregations"
  88. Under "Fields", find "fake_x" and click on the three-dots to the right of its label. A dropdown menu should appear
  89. Click/select "Hide". The "fake_x" field should darken/grey-out. Now when hovering over each distinct bar in the bar chart you will see a tooltip showing the value and the corresponding aggregation function, as shown in the figure below:
  90. Bar Chart Tooltip Example
  91. Close the "Format visual" section by clicking the "x" button next to the name at the top of the section. The "Fields list" should re-appear
  92. In "Fields list", hover over "total_avg_response_time" until a vertical three-dots button appears to the right of the label. Click on it. A dropdown menu should appear
  93. Hover over "Format: 1,234.57". A dropdown menu should appear to the right
  94. Click on "More formatting options...". The "Fields list" section should be replaced with "Format data"
  95. Un-collapse the "Units" section
  96. In the input field labeled "Suffix", enter " ms". Do not forget the leading space. You should see the values in the vertical bar chart for "total_avg_response_time" update to include the new "ms" suffix
  97. Change the bar chart's title to "Total Average Response Time"
  98. Change the bar chart's subtitle to "Minimum, Median, Average, Maximum"
  99. Duplicate the bar chart following the instructions for duplicating charts above
  100. For this new chart, change all of its aggregated values from "total_avg_response_time" to "total_reqs_per_second". You may need to change the aggregation function from "Count" to "Min"/"Median"/etc. after changing the value's field
  101. Change this new chart's title to "Total Requests Per Second"
  102. Ensure this new chart's subtitle is "Minimum, Median, Average, Maximum"
  103. Duplicate this new chart
  104. Change all aggregated values to "total_requests"
  105. Change the "total_requests" chart's title to "Total Requests"
  106. Ensure this new "total_requests" chart's subtitle is "Minimum, Median, Average, Maximum"
  107. Resize and move the 3 bar charts such that they are arranged at the top of the dashboard in a similar way to the following figure:

  108. Vertical Bar Chart Arrangement

  109. Next, recreate the comparison results pie chart

    1. Create a new visual following the steps outlined above
    2. In "Visual types" choose "Pie chart"
    3. Set the "Group/Color" in "Field wells" to "compare_result". The pie chart should update and show the ratios of each type of comparison result
    4. Rename the bottom-most label from "Group By: compare_result" to "Comparison Result"
    5. Change the title to "Comparison Results"

  110. Finally, recreate the percentiles and average response time bar charts

    1. Add a new visual and set its type to "Vertical bar chart"
    2. Set the x-axis to "date_time"
    3. Set the values to "total_95_percentile", "total_99_percentile", "total_100_percentile". Ensure each value's aggregation function is "Sum". The bar chart should update and show each response time percentile for each date
    4. Following the instructions outlined above, set "date_time"'s sort to "raw_timestamp" so that the graph is sorted by time
    5. Following the instructions outlined above, enable data labels such that the value of each bar in the bar chart is displayed above the bar
    6. Following the instructions outlined above, for each field in "Values" add a " ms" suffix to the field. The values displayed in the bar chart should now include a "ms" suffix to indicate that the value is a measure of time
    7. Change the chart's title to "95%, 99%, and 100% Total Response Time Percentiles over Time"
    8. Change the x-axis's title to "Date and Time (EST)"
    9. Duplicate the chart
    10. Change the chart's values to be "total_min_response_time", "total_median_response_time", and "total_max_response_time". Ensure each value's aggregation function is "Sum", not "Count". The bar chart should update and show each response time for each date
    11. Following the instructions outlined above, change the suffix for "total_min_response_time" and "total_median_response_time" to " ms"
    12. Change the chart title to "Minimum, Median, and Average Response Times (ms) over Time"
    13. Ensure the x-axis title is "Date and Time (EST)"

  111. Click on "ADD" in the top-left

  112. Select "Add title"
  113. Set the sheet's title to "server-regression Statistics from TEST Environment 'master' Deployment Runs"
  114. On the top-right of the screen, click the share icon (next to the save and download icons). A dropdown menu should appear
  115. Click "Publish dashboard"
  116. Name the dashboard "bfd-test-server-regression"
  117. Click "Publish dashboard". A new page should load showing the completed Dashboard
  118. Click the share icon on the top-right of the screen. A dropdown menu should appear
  119. Click "Share dashboard". A new page should load with sharing options
  120. In the bottom right, under "Enable access for", select "Everyone in this account"
  121. Select "Discoverable in QuickSight" as well
  122. Close the Dashboard and return to the QuickSight "home" by clicking on "QuickSight" on the top-left of the screen
  123. Return to the "Analyses" view
  124. Open the "bfd-test-server-regression" analysis
  125. When loaded, click on the save icon on the top-right of the screen. A dialog pop-up should appear titled "Save a copy"
  126. In the input field, enter "bfd-prod-sbx-server-regression"
  127. Click "SAVE". The copied Analysis should load
  128. In the left pane, click the pencil icon next to the "Dataset" label. A dialog pop-up should appear with the title "Datasets in this analysis"
  129. Click on the vertical three-dots. A dropdown menu should appear
  130. Click on "Replace". A new dialog should appear named "Select replacement dataset"
  131. Select the "bfd-prod-sbx-server-regression" Dataset
  132. Click "Select"
  133. Click "Replace". All graphs in the Analysis should update appropriately with the new Dataset's field values
  134. Change the sheet's title to "server-regression Statistics from PROD-SBX Environment 'master' Deployment Runs"
  135. Repeat the steps outlined above to publish and share the Analsyis as a Dashboard named "bfd-prod-sbx-server-regression"
  136. Repeat the previous steps to duplicate the "bfd-prod-sbx-server-regression" Analysis. Name the duplicated analysis "bfd-prod-server-regression"
  137. Repeat the previous steps to replace the "bfd-prod-sbx-server-regression" Dataset with "bfd-prod-server-regression"
  138. Change the sheet's title to "server-regression Statistics from PROD Environment 'master' Deployment Runs" following instructions outlined above
  139. Repeat the steps outlined above to publish and share the Analsyis as a Dashboard named "bfd-prod-server-regression"

Once the steps above are complete, there should be three new Dashboards available to view in QuickSight displaying performance metrics for each of the three environments (TEST, PROD-SBX, and PROD). These Dashboards should appear similar to the following:

Final Dashboard Preview