• Skip to primary navigation
  • Skip to main content
The Data Lab

The Data Lab

  • Business Support
        • Business Support

          We’ll help you harness the power of data so you can innovate and grow your business.

          Visit our Business Support page

        • Accessing Talent
          • Data Talent
          • Placements
        • Funding
        • Small Business Support
        • Digital Strategy
        • Academic Project Funding
        • The Data Lab Community
  • Professional Development
        • Professional Development

          We’ll help you harness the power of data so you can innovate at work and also advance your career.

          Visit our Professional Development page

        • Workshops
        • Online Courses
        • Data Skills for Work Programme
        • The Data Lab Community
  • Students
        • Students

          We’ll help you learn about the power of data and gain real-world experience and career-focused qualifications.

          Visit our Students page

        • The Data Lab Academy
        • PhD
        • TDL Academy Placements
        • Scholarships
        • The Data Lab Community
  • Partner With Us
        • Partner With Us

          We work in partnership with companies to help them gain maximum benefit from the strategic use of data.

          Visit our Partner With Us page

        • Collaborate With Specialists
        • Partnership Opportunities
  • About Us
        • About Us

          We discover opportunities, connect people and ideas, develop knowledge and expertise and bring game-changing data projects to fruition.

          About Us

        • Our Team
        • Careers With Us
        • Join our board
        • Academic Opportunities
        • The Data Lab Community
        • Case Studies
        • News & Podcasts
        • DataFest
        • Scottish AI Alliance
        • Contact us

openSIMD: Opening up the Scottish Index of Multiple Deprivation

News 15/09/2017

by Maike Waldmann and Roman Popat

—

You have heard of the Scottish Index of Multiple Deprivation (SIMD) but wonder how it was calculated? You want to use SIMD but without all the health data in it? You want a rich dataset to play with in R? You want to calculate your own composite index but don’t know where to start? openSIMD is the solution to all your problems!

openSIMD makes the calculation steps between the indicator data and the final SIMD measure completely transparent and open to scrutiny “ no black box any more.

openSIMD = R code and documentation

Apart from being the solution to all your problems, openSIMD is a bit of R code along with documentation and data, which lets you calculate SIMD16 for yourself while making any changes you want.

The R code consists of one script to calculate the SIMD domains, another script to calculate the overall SIMD, and a third script with some functions. The documentation explains how to run the code, what the functions do, and how well the code replicates the original code that was used to calculate the official SIMD16. The data consists of two datasets downloaded from the SIMD webpages: The SIMD16 indicator dataset, and the SIMD16 domain ranks dataset.

SIMD identifies Scotland’s most deprived areas

SIMD is the Scottish Government’s official tool for finding the most deprived areas in Scotland. SIMD is used by government, councils, charities and communities as evidence to help target their work to those areas that need it most. SIMD is best known for how it ranks each small area in Scotland by how deprived it is. But in addition to the rankings, all indicator datasets that go into SIMD are also published on a small area level. This data provides a wealth of detailed information about the underlying issues in deprived areas.

SIMD is made up of over 30 indicators which are grouped into seven domains of deprivation. Each domain summarises one aspect of deprivation by combining some of the indicators and using the resulting domain scores to rank each area in Scotland. The seven domain rankings are then combined into an overall, multiple-deprivation SIMD ranking.

Collaborating on openSIMD

The openSIMD project was a collaboration of The Data Lab and the Scottish Government. Maike from the Scottish Government SIMD team says about the project:

When calculating SIMD16, I looked at the SAS code accumulated over the years by my predecessors, and found that it could do with some reviewing and tidying up. At the same time, a colleague put me in touch with Roman from the Data Lab, and we came up with this little project where we would translate the SAS code into R and make it public. I saw it as an opportunity to do some R, promote the use of R at my work, and also as a great way of demystifying SIMD. Another benefit is that we can now point people who want to do complicated things with SIMD to openSIMD and make their and our own lives much easier.

Roman from The Data Lab says:

I was thrilled about the opportunity to help open up such a high profile statistical product. I am a strong believer in open source and I think the current strength and reach of data science is owed to the dominance of open source tools and the OS community. Opening SIMD will be good for transparency but also reproducibility and therefore progress in research. And R users, yes there is a package in the works. If you want to collaborate, get in touch. Also check out the indicators dataset, extremely rich.

Some technical details

We translated SIMD to openSIMD from SAS to R. You can find our documentation for this project here. If you want to fork and contribute to the project, we would be delighted. Please see the public GitHub repository here. If you click through to read the documentation, run the code or explore the results, you will notice that SIMD and openSIMD scores and ranks are not numerically identical. Our tests showed that this is due to the exact way that some algorithms are implemented between the two platforms.

The functions that we have defined in the project are designed to be very SIMD specific. This was to keep us on the straight tracks of the SIMD procedure and not to create new more general tools. Finally, we realise that the fundamental unit of repeatable analysis in R is the package. For practical reasons we decided against writing a package in the first instance, however we plan to convert the project into a package in due course. If you want to collaborate on this please get in touch.

Find openSIMD

  • openSIMD on GitHub
  • openSIMD on the Scottish Government website

Feedback

If you have questions or want to give feedback, there are two ways to do it. For any questions about SIMD, please contact the SIMD team at simd@gov.scot. For any questions about the technical implementation in R, please raise an issue on the GitHub repository. We look forward to interacting with you.

Enjoy openSIMD!

Innovate • Support • Grow • Respect

Get in touch

t: +44 (0) 131 651 4905

info@thedatalab.com

Follow us on social

  • Twitter
  • YouTube
  • Instagram
  • LinkedIn

The Data Lab is part of the University of Edinburgh, a charitable body registered in Scotland with registration number SC005336.

  • Website Accessibility
  • Privacy Policy
  • Terms & Conditions

© 2023 The Data Lab

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsReject AllAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-advertisement1 yearSet by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
CookieLawInfoConsent1 yearRecords the default button state of the corresponding category & the status of CCPA. It works only in coordination with the primary cookie.
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
_ga2 yearsThe _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga_DPXX4XJSJ82 yearsThis cookie is installed by Google Analytics.
_gat_gtag_UA_54851888_11 minuteSet by Google to distinguish users.
_gat_UA-54851888-11 minuteA variation of the _gat cookie set by Google Analytics and Google Tag Manager to allow website owners to track visitor behaviour and measure site performance. The pattern element in the name contains the unique identity number of the account or website it relates to.
_gcl_au3 monthsProvided by Google Tag Manager to experiment advertisement efficiency of websites using their services.
_gid1 dayInstalled by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
CONSENT2 yearsYouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
CookieDurationDescription
personalization_id2 yearsTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.
VISITOR_INFO1_LIVE5 months 27 daysA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSCsessionYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devicesneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-idneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
CookieDurationDescription
cl-bypass-cache1 hourNo description
muc_ads2 yearsNo description
SAVE & ACCEPT
Powered by CookieYes Logo