SODU - Code The City

Posts about SODU written by ianwatt. ... We may have an existing Open Data Strategy for Scotland (2015) stating that Scotland's data is “open by default”, ... Skiptocontent About Whatwedo Hackweekends LunchandLearnSessions PythonUserGroup CodeofConduct Contact SearchSearchEventsArticlesArchives October2022 September2022 January2022 December2021 November2021 September2021 July2021 June2021 March2021 December2020 September2020 August2020 July2020 June2020 May2020 April2020 March2020 January2020 December2019 November2019 September2019 June2019 May2019 January2019 December2018 September2018 June2018 January2018 December2017 November2017 October2017 September2017 February2017 January2017 November2016 October2016 September2016 August2016 May2016 March2016 January2016 November2015 October2015 June2015 May2015 February2015 October2014 June2014 TheScottishOpenDataUnconference(SODU)isbackforathirdyear. Sinceourinceptionin2014CodeTheCityhasconsistentlychampionedOpenData.OurtrusteessetupODIAberdeen,theonlyScottishnodeoftheOpenDataInstitute.Asanorganisation,andasindividualtrustees,we’veworkedtohighlightthecasetogovernmentforopendata,touseopendata,tomakedataopen,andtoeducateothersinthebenefitsandopportunitiesofopendata.Thehighly-successfulOpenDataScotlandportal,builtbyateamofcommunityvolunteerstoaddressagapinGovernmentprovision,grewfromconversationsatpreviousSODUeventsandhackweekendswhichwehosted. Whyopendataisimportant Makingdata,particularlygovernmentdata,openhasverymanybenefits–fromtransparencytocitizenempowerment;fromprovidingfuelforinnovationtoimprovinggovernmentefficiencyandmuchmore.TheOpenDataHandbookisagreatsourceoffurtherbackgroundonthis.Inearly2020theEUDataportalpublishedameta-studyoftheeconomicvalueofopendatainearly2020.FromthisweextrapolatedthattheeconomicpotentialtoScotland,ifODpublishingweredonewell,couldbeinexcessof£2billionperannum.Ourtrustee,IanWattpublishedaresearchpaperWhatisOpenDataandWhyDoesitMatter?withtheDavidHumeInstitutewhichisavailablehereunderanCC-VY-SA4openlicence. AbriefhistoryofSODU Backin2020wecameupwiththeideaofaScottishOpenDataUnconference–aneventwhichanyonewithaninterestinopendatacouldattend(whetheranactivistormildlycurious;whetherinGovernment,civicsociety,academia,orindustry).TheeventwasinpartinspiredbyourhostingUKOpenDataCampin2018butourintentionwasthatitwouldtargetourgrowingnetwork.Itwouldn’tjustbeaboutODinScotlandbutwouldlookworldwidetohelpScottishattendeesdeveloptheirunderstandingofOD,createsmallnetworkstoaddresschallengeslocally,helppublishersconnectwiththeirdataconsumersandviceversa. ThenCovidgotintheway–andourMarch2020event(forwhichwehad107attendeessignedup)waspushedbackandrunasanonlineeventinSeptemberwithjustathirdofthatnumber.WerepeateditonlineagaininOctober2021withasimilarnumberofattendees. We’renowdelightedthatSODU2022isbackasaphysicaleventattheONETechHub,Aberdeenwhichisagreatspaceformeetingandworkingtogether.We’llalsohavehighqualitycateringtolookforwardto!Andwe’redoingallwecantomakeitassafeaspossiblehere. Whattoexpect BasedonpreviousattendanceatSODU2020and2021,andsimilareventswe’verun,weexpectabroadmixofattendees.Therewillbeastrongpresencefromcivicsociety,withothersfromacademia,thepressandtheIT/Dataindustry.Warealsohopefulthatwewillseesomeattendeesfromgovernment,giventheirpolicy/strategy/legislativeobligationtopublishdata. Eachdaywillbedifferent,dependingonwhoisthereandwhattheywanttodiscuss(seeformatbelow).Somewillattendtoshowtheirworkorshareresearch;otherstorunpracticalsessions.Somewillofferinformaleducationalslots;otherswillseeksupportforprojects;andothersstillwillseekorprovidefeedbackonplans,ordataprovision. Thekeytoasuccessfuleventisthemixofattendeesandpeoplebringingapositive,openattitude,andawillingnesstoengage. Format Thiseventwillfollowtheopenspaceformat,whereparticipantscreatetheagendaeachmorningforthesessionsthatwillhappenthatday.Thisisthetimetopitchyourquestionstothemanypeoplewhowillbethere,andtotellothersaboutyourprojecttoo. Sessionsinopenspacealwaysworkbestwhenthereisaninteractiveaspectsothattheparticipantstalkasmuchasthepersonchairingthesession.Thisshortvideowilltellyoumoreaboutopenspace.Theamazingthingisthatitscalesreallywellfrom20peopletoover1000people. Booking We’veseveraltypesofticketsavailable. Weekend-longonesforstudentsatjust£15, Generalweekendticketsforjust£25, Dailyticketsat£15foreithertheSaturdayorSunday,and Weekendonlineticketsforjust£10. Andifthepriceisabarriertoyourattendancejusttellusandwe’llsortsomethingoutforyou. Theseeventsworkbestwithabroadmixofattendees.Nomatterwhatyourskillset,interestareaorknowledgethereissomethingforyou–andsomethingyoucancontribute.Thewrite-upsofthe2020andthe2021eventswillremindyouofhowmuchcanbeachievedbyagroupofdedicatedpeopleoverjustafewhours! Bookhere. Wehopethatyoucanjoinus. Ian,Karen,Andrew,Pauline,BruceCodeTheCityCharitySC047835 AguestpostbyKarenJewell,aDataScientistwhoattendedSODU2020.  IwentintotheweekendofSODU,headsetandcoffeeattheready,thinkingthatasSODUwasbothmyfirstexperienceofanunconferenceandofOpenData,Iwouldn’tbeabletoparticipatemuchbutthatIcouldtaketheopportunitytolearnfromthebrighterandmoreinformedvoicesaroundme.Well,itturnsoutIwasquitewrongaboutmyinvolvementwithOpenData. Inthenetworkingsessionsofthefirstday,Iintroducedmyselfassomeonewhodidn’tworkatallwithOpenData,hadnoexperienceofitandwasheretolearnaboutit.Yetastheeventcarriedonthroughtheday,manydiscussionsandconceptsseemedfamiliartomeandintheafternoonofthefirstdayIhadthat“ah-hah!”moment.Irealiseditwasn’ttruethatIdidnotworkwithOpenData,Idid,andactuallyhaddonesoquiteabitinthelast12months.Ijusthadnotrealisedthatiswhatitwascalled. OpenDataisdatawhichisnotownedorcontrolled,andisfreeforuseanddistribution.HavingonlyjustcompletedmystudiesinaMScDataScienceattheRobertGordonUniversity3weeksprior,freedatawasprettycriticaltomyworkasastudent.NotonlywasIabletopracticeconceptsusingfreelyavailabledatasets,3ofmy8taughtmodulesrequiredmetosourcemyowndatasetforthatmodule’sassessment.Torephrasethat,IneededOpenDatatocompletemydegree.DataScientistsareawareofKaggle,theUCIMLrepository,andaquickonlinesearchforScotland’sdatawillreturntheScottishGovernment’sstatisticsportal.Weseethesesourcesasfreedatawecanpracticeon,butwemaynothaverecogniseditasOpenData,Icertainlydidn’tuntilSODUtookmyblinkersoff. ComingoutofSODU,Istartedtowonderhowmanyotherpeoplewereinthesamemetaphoricalboat.WeretheynotansweringthecallforinvolvementbecausetheydidnotrealisetheavailabilityofOpenDataaffectedthemtoo?Totesttheidea,Isetupanon-scientificsurveyonInstagramandaskedmypeersthequestion“DoyouknowwhatOpenDatais?”withasimple“Yes/No”response.Ofthe22personswhoresponded,3saidYes(14%)and19saidNo(86%).Inaperfectworld,Iwouldhavealsohadafollow-upquestionaskingiftheyhadusedinformationfromalistofknownOpenDatasourcestoconfirmthetheory,butwewillhavetodowithoutfornow. QuickPoll YetintheageofCovid-19whereeveryoneisquitecapableofquotingastatisticormethodineveryonlineargumentforandagainst,howmanyofushaven’trealisedwearebenefitingfromtheavailabilityofOpenDatawhenwequotenewcasecounts,%positivetests,andinfectionratesinourconversationsonadailybasis? IattendedSODUtolearnaboutOpenData,andIlearntI’dactuallybeenusingitallalong.SeveralprominentthemesdiscussedatSODUincludedtheneedforacommunityofpractitioners,havingacentralpointofaccess,andhavingevidenceofthebenefitsofsupportingOpenData.Thequestionthatbugsmenowis,howdoweknowwhoourpractitionersandwhereoursuccessstoriesare,iftheycan’tevenrecognisethemselves?Maybe,thereisanopportunitytodosomeworkhere?   “Hearmecalling,hearmecallingloud,  Ifyoudon’tcomesoon,I’llbewearingashroud.”–TenYearsAfter(1969) Introduction TodaymarksthetenthanniversaryofmyinvolvementwithOpenDatainScotland. AsIwrotehere,backin2009-2010I’dbeenfollowingtheworkthatChrisTaggartandothersweredoingwithopendata,andwasinspiredbythemto createwhatInowbelievetohavebeenthefirstopendatapublishedinthepublicsectorinScotland. @CountCulturecanyouaddtoOpenlyLocalplease?http://bit.ly/bxGNmrIbelieve@AberdeenCCis1stScottishLAtopublishopendata —IanWatt(@watty62)August20,2010 Thispieceisareflectionofmyownviews.TheseviewsmaybethesameasthoseheldbycolleaguesatCodeTheCityorindeedonthecivicsideoftheOpenGovernmentPartnership.I’venotspecificallyaskedotherindividualsineithergroup. Whilemyinvolvementin,andchampioningof,opendatainScotlandisnowadecadelong,myenthusiasmforthesubjectandinthethesocialandeconomicbenefitsitcandeliver,isundiminishedbymyleavingthepublicsectorin2017afterthirtyfouryears.Infacttheoppositeistrue:themoreIaminvolvedintheODmovement,andstudywhatisbeingachievedbeyondScotland’snarrowborders,themoreIamconvincedthatweareacountryintentonsquanderingarichopportunity,regardlessofourpoliticians’publicpronouncements. Butthejourneyhasnotbeeneasy.primarilyduetoalackofdirectionfromScottishGovernmentandlittlecommitment,resourceorengagementatalllevelsofpublicservice.AfriendwhoreviewedthisblogpostsuggestedthatIshouldreplacethepictureofabirthdaycake(above)withoneofanakedhumanbackbearingbleedingscarsfromtheourbattles.He’sright– itisSTILLabattletenyearson. ItisnotasifthepositioninScotlandisgettingbetter.Wearemovingataglacialpace.ThegapbetweenScotlandandothercountriesinthisregardiswidening.IgaveatalkearlierthisyearinwhichIshowedassessmentsofScotland,RomaniaandKenya’sperformanceinOpenGovernment(source:https://www.opengovpartnership.org/campaigns/global-report/Vol2)andaskedtheaudiencetoidentifywhichwasScotland. ExtractsfromVol2tooftheOpenGovPartnershipreport Showfullversionofgraphic Question:WhichisScotland?(Answer) Economicopportunities InFebruary2020theEuropeanDataPortalpublishedareport–TheEconomicImpactofOpenData–whichsetsoutacleareconomiccaseforopendata.Thatpaperlooksat15previousstudiesbetween1999and2020whichhaveexaminedatthemarketsizeofopendataatnationalandinternationallevels,measuredintermsofGDPofeachstudy’sgeographicalarea. Takingtheaverageandmedianvaluesfromthosereports(1.33% and1.19%respectively)andanestimatedGDPforScotland(2018)of£170.4bnwecanseethatthemissedopportunityforScotlandisoftheorderof£2.027bnto£2.266bnperannum. WhatistheactualvalueofthelocalmarketcreatedbyScottish-createdopendata?ifpushedforafigureIwouldestimatethatitiscurrentlyworthafewhundredthousandpoundsperannum,andnomore.Quiteagap! Meantimewehavetheusualsuspectof consultants whisperingsweetlyintheearsofministers,seniorcivilservantsandcouncilbossesthatweshouldbemonetisingdata,creatingmarkets,sellingit.Therewillbenomention,Isuspectoftheheavily-subsidised,privatesectorled,yetfailedCopenhagenDataExchange,Isuspect.(Maybetheycanmakeafewbobbacksellingthedomainname!) YoucanbuythefailedCityDataExchange.comforjust$5195 WhilethiscommercialapproachtodatamayplugsmallgapsinannualfundingforScotland,andlinethepocketsofsomebigcompaniesintheprocess,itwon’tdeliverthefinancialbenefitsatanationallevelofanythinglikethefiguressuggestedbythatEUDataPortalreportbutitwill,intheprocess,activelyhamperinnovationandinhibitsocietalbenefits. Ihearlotsofinstitutionssaying“weneedtoselldata”or“weneedtosellaccessrightstothesephotos”orsimilar.Yet,insomanycases,theoperationofthemechanismsofcontrol;thestaffing,administration,paymentprocessingetc.faroutstripsanygeneratedincome.WhenIchallengedexcolleaguesinlocalgovernmentaboutthisbehaviourtheirresponsewas“butourmanagerswanttoseeanincomeline” towhichwecouldadd“nomatterhowmuchitiscostingus.” AndthistweetfromTheFerretonTuesdayofthisweekisanotherexcellentexampleofthis! Ihavealsoheardlotsofpoliticalproclamationsof“openandtransparent”governmentinScotlandsince2014.Yetmostoftheevidencepointsinexactlytheoppositedirection.Don’tforget,whenCovid19struck,Scotland’sgovernmentwasreportedlytheonlypoliticaladministrationapartfromBolsonaro’sfarrightoneinBraziltousetheopportunitytolimitFreedomofInformation. Openness,really? ItisclearthatthereislittleornocommitmenttoopendatainanymeaningfulwayataScottishGovernmentlevel,inlocalauthorities,oramongnationalagencies.Thisisnottosaythattherearen’tcivilservantswhoaredoingtheirbest,oftenfightingagainstpoliticalorsenioradministration’sactions. Publicdeclarationsarerarelymatchedbydeliveryofanythingofsubstanceandconversationswithpeopleinthoseagencies(ofwhichIhavehadmany)paintsagrimpictureofpoliticalmasterssayingonethinganddoinganother,ofseniormanagementnotbackinguppublicstatementsofintentwiththenecessaryresourcecommitmentand,onmorethanoccasion,suggestionsofbadactorsactuallygoingagainstwhatisofficialpolicy. ImentionbelowthatIjoinedtheOpenGovernmentPartnershiplatein2019.InitiallyIwasenthusiasticaboutwhatwemightachieve.Whiletherearecivilservantsworkingdedicatedlyonopengovernmentwhowanttomakeitwork,Iamunconvincedaboutpoliticalcommitmenttoit.WereallyneedtogetsomepositiveandpracticaldemonstrationthatScottishGovernmentarebehindus–otherwiseIandtheothercivilsocietyrepresentativesarejustassistinginanopen-washingexercise. Inmyview(andthatofothers)thepressinScotlanddoesnotprovideadequatescrutinyandchallengeofgovernment.Wehavearemarkablyineffectivepoliticalopposition.WealsohaveanetworkofagenciesandquangoswhicharereliantontheScottishGovernmentforfundingwhoareunwillingtopushback.Allofthisgivesthepoliticalsideafreepasstospoutencouragingwordsof“openandtransparent”yetdotheminimumatalltimes. WemayhaveanexistingOpenDataStrategyforScotland(2015)statingthatScotland’sdatais“openbydefault”,yet my2019calculationwasthatover95%ofthedatathatcouldandshouldbeopenwasstilllockedup.Andthereislittlemovementonfixingthat. Wehavemanyexamplesofagenciesdoingonethingandsayinganother,suchas ScottishEnterpriseextollingthevirtuesof OpenDatayetproducingnone.ItsoneAPIhasbeenbrokenformanymonths,Iamtold. MygoodfriendsatTheDataLabdoamazingworkonfundingMScandPhdplaces,andprovidingfundingforindustrialresearchintheapplicationofdatascience.Theirmissionis“tohelpScotlandmaximisevaluefromdata…”yettheycurrently offernoguidanceonopendata,notargetedprogrammeofsupport,nochampioningofopendataatall,despitethewidely-acceptedeconomicadvantageswhichitcandeliver.ThereisthepotentialforTheDataLabtoleadonhowScotlandmakesthemostofopendataandtoguidegovernmentthinkingonthis! Allofthisisnottopickonspecificorganisations,orhardworkinganddedicatedemployeeswithinthem.ButitdoeshighlightsystemicfailuresinScotlandfromthetopofgovernmentdownwards. Fixingthisisanenormoustask:onewhichcanonlybedonebythedevelopmentofafreshstrategyforopendatainScotland,whichismandatedforallpublicsectorbodies,isfundedasaninvestment(recognisingtheeconomicpotential),andwhichisrigorouslymonitoredandenforced. Icouldgoon….butlet’slookatthisyear’ssurvey. (skiptosummary) Anotheryearwithwhattoshowforit? InFebruary2019IconductedasurveyofthestateofopendatainScotland.Itdidn’tpaintanencouragingpicture.Thedatabehindthatsurveyhasbeenpreservedhere.Ayearon, Istartedthinkingaboutrepeatingthereview. IntheinterveningyearI’dbeeninvolvedinquiteabitactivityaroundopendata.Ihad joinedthecivicsideoftheOpenGovernmentgroupforScotlandandwasaskedtoleadfor thenextiterationoftheplanonCommitmentThree(sharinginformationanddata), joinedthesteeringgroupofStirlingUniversity’sresearchproject,DataCommonsScotland, trainedasatrainerforWikimediaUK,deliveringtraininginWikidata,WikipediaandWikiCommons,andrunningmultiplesessionsforCodeTheCitywithafocusonWikidata, createdanopenSlackGroup fortheopendatacommunityinScotlandtoengagewithoneanother, createdanOpenDataScotlandtwitteraccountwhichhasgainedalmost500followers,and initiatedthefirst ScottishOpenDataUnconference(SODU)2020whichhadbeenscheduledtotakeplaceasaphysicaleventinMarchthisyear.Thathasnowbeenreconfiguredasanonlineunconferencewhichwillhappenon5thand6thSeptember2020. Inrestartingthisyear’sreviewofopendatapublishinginScotlandmyaimsweretoseewhathadchangedintheintervening12monthsandtoincreasethecoverageofthesurvey:goingbroaderanddeeperanddevelopinganevenmoreaccuratepicture.ThatworkspilledintoMarchatwhichpointCovid-19struck.DuringlockdownIwasdistractedbyvariouspiecesofwork.Itwasn’tuntilAugust,andwithagrowingsenseoftheimminenceofthis10-yearanniversary,thatIwasgalvanisedtofinishthatreview. Iamconsciousthatthemethodologyemployedhereisnotthecleverest–onepersoncountingonlythenumbersofdatasetsproduced. ThisissomethingIreturntolater. Thepicturein2020 Ibrokethereviewdownintosectoralgroupingstomakeitmoremanagabletoconduct.BystickingtothatIhopetomakethisoverviewmorereadable.TheupdatedGitHubrepo inwhichInotedmyfindingsisavailablepublicly,andIencourageanyonewhospotserrorsoromissionstomakeapullrequesttocorrectthem.EachheadingbelowhasalinktotheGithubpagefortheresearch. Overallthereislittlesignificantpositivechange.Thisisonefactorwhichgivesrisetoconcernsaboutgovernment’scommitmenttoopennessgenerallyandopendataspecifically;andtoagrowingcynicismintheciviccommunityaboutwherewegofromhere. LocalGovernment (Sourcedatahere) IreviewedthisareainFebruary2020andrecheckeditinAugust. SadlytherehasbeennosignificantchangeinthepublicationofopendatabylocalgovernmentintheeighteenmonthssinceIlastreviewedthis.Morethanathirdofcouncils(13outofatotalof32)stillmakenoopendataprovision. WhilethebiggainisthatRenrewshireCouncilhavelaunchedanewdataportalwithoverfiftydatasets,mostcouncilshaveshownlittleornochange. SadlytheHighlandCouncilportal,procuredaspartoftheScottishCitiesAlliance’sDataClusterprogrammeat£10,000’scost,hasvanished.Idontthinkiteversawadatasetbeingaddedtoit.SearchingHighlandCouncil’swebsiteforopendatafindsnothing. Whilebignumbersofdatasetsdon’tmeanmuchbythemselves,theCityofEdinburghCouncilhasamighty236datasets.Brilliant!BUT…noneofthemareremotelycurrent.ThelastupdatetoanyofthemwasSeptember2019.Over90%ofthemhaven’tbeenupdatedsince2016orearlier. SimilarlyGlasgow,whichhas95datasetslistedhaveaportalwhichisrepeatedlyofflinefordaysatatime.Aportalwhichwon’tloadisuseless. Dundee,PerthandStirlingcontinuetodowell.Theirofferingsaregrowingandtheydemonstratecommitmenttothelong-haul. Aberdeenlaunchedaportal,morethanthreeyearsintheplanning,populateditwith16datasetsandimmediatelylettheiropendataofficerleaveattheendofashort-termcontract.Someoftheirdatasetsareinterestinganduseful–buttherewasnoconsultationwiththelocaldatacommunityaboutwhattheywouldfinduseful,ordeliverbenefitslocally;alldespitemultipleinvitationsfrommetointeractwiththatcommunityatthelocaldatameet-upswhichIwasrunninginthecity. ItwashopedthattheprogrammeundertheScottishCitiesalliancewouldyielduniformdatasets,prioritisedacrossallsevenScottishCities,butthereisnosignofthathappening,sadly.Sowhatyoufindonallportalsorplatformsisprettymuchapot-luckdraw. Wherecommonstandardsexist–suchasthe360Givingstandardforthepublicationofsupportforcharities–organisationsshouldbeuniversallyadoptingthese.Yetthisisonlyusedbytwoof32authorities,allofwhomhavegrant-makingservices.Surely,duringapandemicespecially, itwouldbeadvantageoustofundersandrecipientstoknowwhoisfundingwhichbodytodeliverwhatproject? Councils–OpenGovernmentLicenceandRPSI Thisisaslightasidefromthepublicationofopendata,butanimportantone.IftheScottishAuthoritiesweretoadoptanOGLapproachtothepublicationofdataandinformationontheirwebsite(asboththeScottishGovernment’scoresiteandtheInformationCommissionerforScotlanddo)thenwewouldbeabletoatleastreusedataobtainedfromthosesites.Thisisnotareplacementforpublishingproperopendatabutitwouldbeatinystepforward. Thetablebelow(sourceandreviewdata here) showsthecurrentpermissionstoreusethecontentofScottishLocalAuthorities’websites.Manyarelackinginclarity,havemessywording,arevagueormisunderstandterminologies.Theyalso,inthemain,ignorelegislationonfairre-use. TableoflocalauthorityadoptionofPGLandRPSI OpenGovernmentLicence TheScottishGovernment’sownsiteisexcellentandclear:permittingallcontentexceptlogostobebereusedundertheOpenGovernmentLicence.Thisisnottrueforlocalauthorities.AtpresentonlyFalkirkandOrkneyCouncils–twoofthesmallerones–allow,andpromoteOGLre-useofcontent.Thereisnogoodreasonwhyallofthepublicsector,includinglocalgovernment,shouldnotbecompelledtoadoptthetermsofOGL. Re-useofPublicSectorInformation(RPSI)Regulations Since2015thepublicsectorhasbeenobligedbytheRPSIRegulationstopermitreasonablereuseofinformationheldbylocalauthorities.So,evenifScottishLAshavenotyetadoptedOGLforallwebsitecontent,theyshouldhavebeenmakingitclearforthelastfiveyearshowacitizencanre-usetheirdataandinformationfromtheirwebsite. InmylatesttrawlthroughtheT&CsandCopyrightStatementsof32ScottishLocalAuthorities,Ifoundonly7referencingRPSIrightsthere,with25notdoingso(seethefulltableabove).Iamfairlysurethattheseauthoritiesarebreakingthelegalobligationonpublicbodiestoprovidethatinformation. Finally,giventhepresenceofCOSLAontheOpenGovernmentScotlandsteeringgroup,thesituationwithnoopendata;poor,missingoroutdateddata;andOGLandPRSIissuesneedstoberaisedthereandsomereassurancesoughtthattheywillworkwiththeirmemberorganisationstofixtheseissues. Health (Sourcedatahere) TheNHSScotlandOpenDataplatformcontinuestobedevelopedasaveryusefulresource.Thenumberofdatasets therehasmorethandoubledsincelastyear(from26to73). NoneofthefourteenHealthBoardspublishtheirownopendatabeyondwhatisontheNHSScotlandportal. OnlyoneofthethirtyHealthandSocialCarePartnerships(HSCPs)publishanythingresemblingopendata:AngusHSCP. COVID-19andopendata Whileweareonhealth,I’vewrote(hereandhere)earlyinthepandemicabouttheneedforopendatatohelpthebetterpublicunderstandingofthesituation,andstimulateinnovativeresponsestothecrisis.ThestatisticsteamatScottishGovernmentrespondedwelltothisandwe’vestartedtodevelopagoodrelationship.I’venotfollowedthatupwitharetrospectiveaboutwhatdidhappen.PerhapsIwillintime. ItwasclearthattheneedforopendatainCV19situationcaughtgovernmentandhealthsectornapping.Theresponsewasslowerthanitshouldhavebeenandpatchy,andtherearestillgaps.Peoplefinditdifficulttolocatedatawhenitisonmulipleplatforms,spreadacrossScotsGovt,HealthandNRS.Thatis,inamicrocosm,oneoftherealchallengesofODinScotland. WithanopenSlackgroupforOpenDataScotlandthereisadirectchannelthatdataproviderscouldusetoengagetheopendatacommunityontheirplansandproposals.Theycouldalsotosoundoutwhatdataanalystsanddatavizspecialistswouldfinduseful.ThatopportunitywasnottakenduringtheCovidcrisis,andwhileIwasOKintheshorttermwithbeingusedasahumanconduittothatgroup,itwasneitherefficientnorsustainable.MyhopeisthatpostSODU2020,andasthenextiterationoftheOpenGovScotlandplancomestogetherwewillseebetter,morefrequent,directengagementwiththedatacommunityontheoutsideofGovernment,andamoreporousborderaltogether. FurtherandHigherEducation (Sourcedatahere) Thereisnosignificantchangeacrossthesectorinthepast18months.Thevastmajorityofinstitutionsmakenoprovisionofopendata.Somehavevagueplans,manyofthemhistoric–goingbackfouryearsormore–andnotactedon. LumpingUniversitiesandCollegestogether,onemightexpectataminimumproperlystructuredandlicensedopendatafromeveryinstitutionon: courses modules events performance(perhapssomeofthisisonHESAandSFCsites?) physicalassets environmentalperformance KPItargetsandachievementsetc. Ofcourse,thereisnoneofthat. Universitiesandcolleges IreviewedopendataprovisionofUniversitiesandCollegesaround17February2020.Irevisitedthison11August2020,makingminorchangestothenumbersofdatasetsfound. Whilefiveoffifteenuniversitiesarepublishingincreasingamountsofdatainrelationtoresearchprojects,mostofwhichareonaCC-0orotheropenbasis,therecontinuestobeaverylimitedamountofrealoperationalopendataacrossthesectorwithloadsofpromisesandstatementsofintent,somegoingbackseveralyears. TheHigherEducationStatisticsAgencypublishesarangeofpotentiallyuseful-lookingOpenDataunderaCC-BY-4.0licence.Thisisdataaboutinsitutions,course,studentsetc–andnotdatapublishedbytheinstitutionsthemselves.ButIcouldidentifynoneofthat.Overall,thiswasverydisappointing. Further,whilethereare20FEcolleges.Noneproducesanythingthatmightbeclassedasopendata.Afewhaveanythingbeyondvaguestatementofintent.PerhapsCityofGlasgowCollegenotonlycomesclosest,butdoeslinktosomesourcesofinfoanddata. TheCrightonObservatory Whiledoingallofthis,IwasremindedoftheCrightonInstitute’sRegionalObservatorywhichwasannouncedtoloudfanfaresin2013andappearstohavequietlybeenshutdownin2017.TwooftheteaminvolvedsayintheirLinkedInprofilesthattheyleftattheendoftheproject.Eventhedomainnametowhicharticlespointisnowupforgrabs(Feb2020). Itnowappears (Aug2020)thatthetotalinitialbudgetfortheprojectwas>£1.1m.Giventhatthepurposeoftheobservatorywastoamassagreatdealofopendata, Ihavealsoattemptedtofindoutwherethedataisthatitcollectedandwheretheknowledgeandlearningarisingfromtheprojecthasbeenpublishedforposterity?Ican’tlocateit.ThisFOIrequestmayhelp.Thebigquestion:whatbenefitsdidthe£1.1m+deliver? ScottishParliament (Sourcedatahere). InFebruary2019IfoundthatTheScottishParliamenthadreleased121datasets.Thiscoversmotions,petitions,Bills,petitionsandotherproceduraldata,andisveryinteresting.Thisyearwefindthattheyhavestill121datasets,so,therearenonewdatasources. Infactthatnumberismisleading. InFebruary2020 Idiscoveredthatwhile75ofthesehavebeenupdatedwithnewdata,theremaining46(markedBETA)nolongerwork.AsofAugust2020thisisstillthecase.Whynotfixthem,oratworstclearthemouttosimplfythefinadbilityofworkingdata? SomeoftheseBETAdatasetsshouldcontainpotentiallymoreinteresting/usefuldatae.g.RegisterofMembersInterestsbutjustdon’twork.Returning:[“{message:‘Dataispresentlyunavailable’}”] Ididn’tnotetheavailabilityofAPIslastyear,butthereare186APIcallsavailable.Manyoftheseareyear-specific.Itestedhalfadozenandaboutathirdofthosereturnederrormessages.Isuspectsomeofthesealignwiththenon-functioninghistoricBETAs. Sadlytheissuesraisedayearagoaboutthelackofclarityofthelicensingofthedataisunchanged.Tofindthelicence,youhavetogotoNotes>PolicyonUseofSPCBCopyrightMaterial.Followingthefirstlinkthere(toaPDF)youseethatyouhavetoadd“ContainsinformationlicencedundertheScottishParliamentCopyrightLicence.”toanythingyoumakewithit,whichisOK.Butifyougotothesecondlink“ScottishParliamentCopyrightLicence”(anotherPDF)thewording(slightly)contradictsthatobligation.ItthenhasachunkaboutOGLbutsays,“ThisScottishParliamentLicenceisalignedwithOGLv3.0”whateverthatmeans.WhynotjustlicenseallofthedataunderOGL?Ican’tseewhattheyaretryingtodo. ScottishGovernment (Sourcedatahere) TryingtoworkoutthebusinessunitswithinthestructureofScottishGovernmentisasignificantchallengeinitself.Attemptingtothenestablishwhichhavepublishedopendata,andwhatthosedatasetsare,andhowtheyarelicensed,isalmostanimpossibletask.Ifmychecking,andarithmeticareright,thenof147discretebusinessunits,only27havepublishedanyopendataand120havepublishednone. Sowecansaywithsomeconfidence thattheissuewithfindabilityofdataraisedinFeb2019isunchanged,therebeingnocentralportalforopendataintheScottishpublicsectororevenforScottishGovernment.SearchingthemainScottishGovernmentwebsiteforopendatayields633results,noneofwhicharelinkstodataonthefirstfourscreenfuls.Ididn’tgodeeperthanthat. TheScottishGovernment’sStatisticsTeamhaveaverygoodportalwith295DataSetsfrommultipleorganisational-providers.Thisisupby46datasetsonlastyearandincludesatwoneworganisations:TheCareInspectorateandRegistersofScotland.Thelatter,sofar(Aug2020),hasnodatasetsontheportal. Therearesomeinterestingnewentrantsintothelistof thosepartsofScottishGovernmentpublishingdatasuchasDavidMacBrayneLimitedwhich is,Ibelieve,whollyownedbySGandistheparent,oroperatorofCalmacFerriesLimited. On1stMarch2020theyreleasedanewdataplatformtogetdataabouttheir29ferryroutes.Thisisverywelcome.Afterchoosingthedates,routesandtraffictypesyoucandownloadaCSVofresults.WhiletheirintentappearstobetomakeitOpenData,thewebsiteiscopyrightandthereisnospecificlicensingofthedata.Thisiseasilyfixable. ItisalsointerestingtocontrastTransportScotlandwithworkgoingoninEngland. TransportScotland’spublicationschemesaysofopendata“OpendatamadeavailablebytheauthorityasdescribedbytheScottishGovernment’sOpenDataStrategyandResourcePack,availableunderanopenlicence.Wecomplywiththeguidanceabovewhenpublishingdataandotherinformationtoourwebsite.DetailsofpublicationsandstatisticscanbefoundinthebodyofthisdocumentoronthePublicationssectionofourwebsite.”IsearchedbothwithoutsuccessforanyOD.Whynotsay“wedon’tpublishanyOpenData”?Comparethiscompleteabsenceofopendatawitheventhesingleproject OpenBusDataforEngland.Readthestoryhere.Scotlandisyet againsofarbehind! Summary InthereviewofdataI’veshownthatlittlehaschangedin18months.Veryfewbranchesofgovernmentarepublishingopendataatall.Thelandscapeislitteredwithoutdatedandforgottenstatementsofgoodintentwhicharenotactedon;brokenlinks;portalsthatvanishordon’twork;outofdatedata;yawninggapsinpublicationandsoon. Theclaimof“OpenByDefault”inthecurrent(2015)OpenDataStrategyismisleadingandmostlyignoredwithconsequence. TheFirstMinistermayfrequentlyrepeatthemantraof“OpenandTransparent”whenspeakingorquestionedbyjournalists,butitiseasilydemonstrablethattheadministrationfrequentlyactinthedirectlyoppositewaytothat. TherecentsituationswithCovid-19andtheSQAexamsresultsshowScotlandwouldhavefounditselfinamuchbetterplacethisyearwithamatureandwell-developedapproachtoopendata:anapproachonemighthavereasonablyexpectedafterfivefullyearsof“openbydefault”. Thesocialandeconomicargumentsforopendataareindisputable.Thesehavebeenacceptedbymostothergovernmentsofthedevelopedworld.Importantly,theyhavealsobeentakenupandactedonbydevelopingnationswhohaveinmanycasesovertakenScotlandintheirdeliveryoftheirOpenGovernmentplans. TheworkIhavedonein2019andinthisreviewisnotasustainableone–i.e.onesinglevolunteermonitoringtheactivityofeverybranchandlevelofgovernment inScotland.Andthemethodologyislimitedtowhatisachievablebyanindividual. AcountrywhichwasseriousaboutOpenDatawouldhavetargetsandmeasures,monitoringandopenreportingofprogress. Itwouldn’tjustcountdatasetspublished.Itwouldbelookingatengagement,theusefulnessofdataanditsintegrationintoeducation. Itwouldfundinnovation:specificallyintheuseofopendata;inthecreationoftools;indevelopingservicestobothsupportgovernmentincreatingdatapipelines,andinhelpingcitizensindatause. Itwouldco-developandmandatetheuseofdatastandardsacrossthepublicsector. Itwoulddevelopandsharecanonicallistsof‘things’withuniqueidentifiersallowingdatasetstobeintegrated. Itwouldadopttheconceptofdataasinfrastructureonwhichnewproducts,services,apps,andinsightscouldbebuilt. IreallywantScotlandtomakethemostoftheopportunitiesaffordedbyOpenData.Iwouldn’thavespenttenyearsatthisifIdidn’tbelieveinthepotentialthisoffers;norifIdidn’thavetheevidencetoshowthatthiscanbedone.Iwouldn’tbegivingupmytimeyear-on-yearresearchingthis,givingtalks,organisinggroupsandcreatingopportunitiesforengagement. WhatisfundamentallylackinghereissomehonestyfromScottishGovernmentministersinsteadoftheirpretenceofsupportforopendata.   IanWatt 20August2020 LinktoanindexofpiecesIhavewrittenonOpenData: http://watty62.co.uk/2019/02/open-data-index-of-pieces-that-i-have-written/ Answertoquiz ScotlandisB,inthecentre.KenyaisA,andRomaniaC. IcouldhavechosenMexico,Honduras,Paraguay,Uruguay–orothers.AllaredoingbetterthanScotland. Backuptothequiz HeaderImagebyDavidBallewonUnsplash. ScottishOpenDataUnconference It’sallgoingoninScotlandinMarch.AswespringintoSpring(nearlythere!),we’reveryexcitedtobesponsoring,andgoing,totheScottishOpenDataUnconferenceinAberdeenon14thand15thMarch.Topicsarepitchedinthemorningofeachday,anagendaiscreatedandparticipantstalkasmuchasthechair.  OurcolleagueJamieWhyteisluckyenoughtohaveaticket,soifyouspothimdosayhi!Herearesomerecentopendatahappeningswe’vepickeduponourradar… ScottishIndexofMultipleDeprivation TheScottishIndexofMultipleDeprivationwasreleasedlateJanuaryandwelovedtheaccompanyingbriefingdocument,whichputthenumbersintocontext(findithere).Thedata’salsoavailableontheScottishGovernment’sOpenDatasite,whereyoucanusetheAtlassectiontofindkeydatazonesandseekeyfactsaboutthem.Thebelowscreenshotisofthedatazonewhichisrankedasthemostdeprivedinthe2020SIMD. SIMD–GreenockTowncentre Peoplearealreadymakingstuffwiththedata — belowisascreenshotofJamie’slavalampvisualisationofthedata.  Commentary,explanationandanalysisfromothersinclude:AlasdairRae’ssummarymatrixoftheSIMDdatabycouncilarea,astorygraphicofthedata,aninteractivemappingtool,ananalysisblogpostfromScottishparliamentinformationcentreandnewsarticles,likethisonefromtheBBC. JamieWhyte–KingoftheLavaLampimage SummarymatrixofScottishIndexofMultipleDeprivation2020,bycouncilarea–shows%ofeachcouncil'sdatazonesbydeprivationdecile–highestdecilehighlightedineachrow(e.g.37%ofEastRenfrewshireinleastdeprived,32%ofInverclyde'sinmostdeprived10%)pic.twitter.com/SwcgBipo51 —AlasdairRae(@undertheraedar)January28,2020 W3CCommunityGroup Anotherthingwe’venoticedisthatthere’spreliminaryworkhappeningonGraphQLandRDF,whichaimstoserveasacaseforfuturestandardisation.Moreonthishere,whereyoucansendarequesttojointhegroupifthisisyourbag.It’sdefinitelyours!  AnewW3CCommunityGrouphasbeenlaunchedtobridgetheworldson#GraphQLand#RDF.Maingoalsareanalysisofexistingapproaches,usecaseandrequirementcollection,toserveasabasisforfuturestandardizationefforts.https://t.co/peTAu0JNO7 —RubenTaelman(@rubensworks)February3,2020 Collaborativeworkwithdata Last,butnotleast,collaboration.Thisisawideconceptbutit’salsoatrendthat’scroppingupindifferentaspectsofworkingwithopendata.Herearesomewe’venoticed: Scotlandisoneof90membersofthecountriestakingpartintheOpenGovernmentPartnership(OGP),andjustoneof15chosentocontributetoitsleadershipandinnovation.ThesecondOGPActionPlanisnowinitsfinalyearandaimsto: “promotetrustandco-operationbetweengovernmentandcivilsociety.” TheOfficeforNationalStatisticsispublishingdatainacollaborativeprojectacrossaspreadoforganisationsincludingONS,HMRC,MHCLG,DWPandDIT.TheConnectedOpenGovernmentStatistics(COGS)projectinvolvesalotoftechnicalcollaborativeworkinharmonisingcodelists,aswellasharmonisingadatamodelandalltheprocessesthatgointoit.MoreonthisprojecthereontheGSSblogsite.  2019sawagrowing,collaborativeAPIcommunity,withAPIeventsinvolvinggovernmentandpeopleworkingwithgovernment.WewenttooneinNewcastleandanotherone’sarrangedforMarch16th(ifyou’restillhungryformoreaftertheunconference!)  TheOpenDataInstitutehavebeenbusy,busy,busy.JeniTennisonspokeabouttheideaofhowcollaborationiskeyfornewinstitutionsofthedataage,atourPowerofDataconferenceinOctober(catchthatvideohere).TheODIhavealsobeenworkingonadataandpublicservicestoolkit&there’sanintroductoryeventtothisinEdinburghjustafewdaysbeforetheScottishOpenDataUnconference.  Thanksforreading!Ifyou’dliketofindoutabitmoreaboutwhoweareandwhatwedo,takealookatourwebsite,ourblog,ourlatestnewsletterand/orourtwitterstream.We’vejustbeennamedasoneoftheFT1000fastestgrowingcompaniesinEuropeandwe’restillhiring,soifyouthinkyoucanhelpuswe’dlovetohearfromyou.  Welovedataandwe’redelightedtobesponsoringtheScottishOpenDataUnconference.Seeyouthere.    Follow Following CodeTheCity Signmeup AlreadyhaveaWordPress.comaccount?Loginnow. CodeTheCity Customize Follow Following Signup Login Reportthiscontent ViewsiteinReader Managesubscriptions Collapsethisbar   LoadingComments...   WriteaComment... Email(Required) Name(Required) Website