– Systematic SIMILAR iterative multilayer literature reviewmethod Journal of Informetrics

(1)

JournalofInformetrics15(2021)101111

ContentslistsavailableatScienceDirect

Journal of Informetrics

j o ur na l h o me p a g e:w w w . e l s e v i e r . c o m / l o c a t e / j o i

Regular article

SIMILAR – Systematic iterative multilayer literature review method

Zsolt T. Kosztyán

^a,c,d,∗

, Tibor Csizmadia

^b

, Attila I. Katona

^a

aDepartmentofQuantitativeMethods,InstituteofManagement,FacultyofBusinessandEconomics,UniversityofPannonia,Hungary

bDepartmentofManagement,InstituteofManagement,FacultyofBusinessandEconomics,UniversityofPannonia,Hungary

cMTA-PEBudapestRankingResearchGroup,Hungary

dInstituteofAdvancedStudies,K ˝oszeg,Hungary

a rt i c l e i n f o

Articlehistory:

Received9July2020

Receivedinrevisedform8October2020 Accepted16November2020

Keywords:

Systematicliteraturereview Multilayernetworks Iterativeclassiﬁcation Citationnetwork Informationsystems

a b s t ra c t

Asthenumberofpublishedscientificarticleshasincreasedexponentiallyandtheinterdis- ciplinarynatureofscientificresearchhasstrengthenedoverthepastdecades,theprocessof conductingefficientliteraturereviewshasplayedanincreasinglyimportantroleinhelping scholarsmakesenseofpreviousresearchresults.Althoughcurrentliteraturereviewmeth- odsprovideinsightfulresults,theyareeithercross-sectionalorlongitudinalstudiesandare unabletosimultaneouslymodelthestructureandevolutionofaresearchfield.Inaddition, onlyafewmethodsapplytheiterativerefinementoftheextractedcategories,andnone integratethepowerfulmultilayernetworktheoryduringtheliteraturereview.Tofillthis gap,thecurrentpaperdevelopsasystematiciterativemultilayerliteraturereview(SIMI- LAR)method.Theproposedmethodhelpsresearchersto(1)refinetheinitialclassification rulesoftheselectedpapersthroughiterations,(2)integratethemultilayernetworktheory intotheliteraturereviewprocess,andfinally(3)conductlongitudinalandcross-sectional analysesatthesametime.WedemonstratetheaddedvalueoftheSIMILARmethodby extendingresearchresultsrecentlyobtainedinthefieldofinformationsystems.

1. Introduction

Themassiveemergenceand rapiddiffusionof knowledgebothwithinandacrossdisciplinesanddomainsrequires researcherstoﬁndawaytoquicklysynthesizepriorresearch,addressrelevantgapsandstimulatefutureresearch.Recog- nizingthatcumulativeknowledgedevelopmentincreasinglyreliesontheintegrationofpreviousstudies,severalscholars havemadecallsformoreliteraturereviews(Bandara,Miskon,&Fielt,2011;Hunter,Schmidt,&Jackson,1982;Paré,Trudel, Jaana,&Kitsiou,2015;Pfeffer&Sutton,2006;Reay,Berta,&Kohn,2009;Rousseau,Manning,&Denyer,2008;Templier

&Paré,2015;Watson&Webster,2020).Conductingeffectiveliteraturereviewsplaysavitalrolenotonlyfortheirown knowledgecontributions,suchasadvancingtheknowledgeandunderstandingthebreadthoftheresearchonatopicof interest,synthesizingknowledge,buildingtheory,aggregatingempiricalevidence,andbecomingorientedinanemerging domain(Cohn&Becker,2003;Cooper,1988;Fink,2019;Mulrow,1994;Paréetal.,2015;Pfeffer&Sutton,2006;Rowe,2014;

∗ Correspondingauthorat:DepartmentofQuantitativeMethods,InstituteofManagement,FacultyofBusinessandEconomics,UniversityofPannonia, Hungary.

E-mailaddress:kzst@gtk.uni-pannon.hu(Z.T.Kosztyán).

https://doi.org/10.1016/j.joi.2020.101111

by/4.0/).

(2)

Schryenetal.,2017),butalsofortheircapacitytostimulatesubsequentresearch,which,inturn,validatesandextendstheir contributions(Paréetal.,2015;Prester,Wagner,&Schryen,2018).

Aliteraturereviewcanbethebackgroundforaprimaryresearcharticle,apartofathesis,apartofaprojectproposal (Baker,2000),oranimportanttypeofpublicationinitsownright(standalonereview)(Schryenetal.,2017;Schwarz,Mehta, Johnson,&Chin,2007;Templier&Paré,2018).Inaddition,reviewarticlesfrequentlybecomecoreormilestonepapersina ﬁeld(Paréetal.,2015).Literaturereviewshavebeenstudiedinseveralﬁelds,suchashealthsciences(Grant&Booth,2009;

Liberatietal.,2009),management(Alvesson&Sandberg,2011;Zorn&Campbell,2006),nursing(Cronin,Ryan,&Coughlan, 2008;Whittemore,2005),psychology(Baumeister&Leary,1997),socialsciences(Hart,2018;Petticrew&Roberts,2008), andsoftwareengineering(Kitchenham&Charters,2007).Unsurprisingly,thediversityofdisciplineshasledtotheadoption ofseveralperspectivesonliteraturereviews,includingdeﬁnitions(Blaxter,2010;Blumberg,Cooper,&Schindler,2005;Fink, 2019;Hart,2018;Rowe,2014;Watson&Webster,2020),purposes(Okoli,2012;Paréetal.,2015;Templier&Paré,2015), typesandclassiﬁcations(Grant&Booth,2009;Paréetal.,2015;Schryenetal.,2017),methodologicalapproaches(Sylvester, Tate,&Johnstone,2013;Wolfswinkel,Furtmueller,&Wilderom,2013)andreviewprocess(Croninetal.,2008;Hart,2018;

Petersen,Feldt,Mujtaba,&Mattsson,2008;Templier&Paré,2018).

Inthisarticle,wefocusonthemethodologicalaspectsoftheliteraturereview.Agrowingnumberofscholarshavecalled forsystematic,well-explicatedandrigorousmethodsthatenhancethevalueofliteraturereviews(e.g.Croninetal.,2008;

Kitsiou,Paré,Jaana,&Gerber,2017;Liu,Lu,Lu,&Lin,2013;Paréetal.,2015;Schryenetal.,2017;Templier&Paré,2018);

suchmethodshelpliteraturereviewstobecomenotonlymoreusefultotheﬁeldbutalsomorereplicable(Templier&Paré, 2018;Wolfswinkeletal.,2013).Attemptsmadeinthisdirectionincludecitation-basedmethods(Liuetal.,2013;Marra, Emrouznejad,Ho,&Edwards,2015;Merigó,Pedrycz,Weber,&delaSotta,2018;Yu,Xu,Pedrycz,&Wang,2017),growth curveanalysis,groundedtheory(Wolfswinkeletal.,2013),thematicanalysis(Bandaraetal.,2011),andthehermeneutic framework(Boell&Cecez-Kecmanovic,2015).

Inaddition,algorithmic-assistedapproachesweredevelopedtosignificantlydecreasethemanualworkwhileconducting theliteraturereview.Brisebois,Abran,Nadembega,andN’techobo(2017)usedmachinelearningtoidentifyandbuildthe relevantliteraturecorpus.Xiongetal.(2018)conductedaliteraturereviewinwhichrelevantpaperswereclassifiedbythe maximumentropyalgorithm.YuandMenzies(2019)developedFAST²,anautomaticassistant,tohelpresearchersfindmore relevantpapersfaster.

Whileprovidinginsightfulresults,thesemethodshavesomeshortcomingsdespitethealgorithmicassistance.Theyare eithercross-sectionalorlongitudinalstudies,andtheydonotallowscholarstomodelthestructureandevolutionofa researchﬁeldatthesametime.Inaddition,currentcitationnetwork-basedsolutionsdonotreﬁnetheextractednetwork structureiterativelyduringtheliteraturesearch,andmultilayercitationnetworkshavenotyetbeenappliedinliterature reviews.

Inthispaper,wefillthisgapbyadvancingastate-of-the-artsystematiciterativemultilayerliteraturereview(SIMI- LAR)methodtohelpresearchersconductliteraturereviewswithouterror-proneandmethodologicalrigor.Specifically, weapplymultilayernetworkstoallowresearcherstoconductbothlongitudinalandcross-sectionalanalysesatthesame time. Furthermore,inthis method,more opportunitiesareavailablesince layerscanbedefinednot onlybythetime dimensionbut alsobasedonvariousfeatures suchasjournalquality orgeographical locationoftheresearchlabora- tory.

Assuch,thispapercontributestotheliteratureinthefollowingways:(1)weadoptanovelcitationnetwork-based literaturereviewmethod,whichallowsresearcherstoiterativelyrefinethepredefinedcategoriesfortheexaminedpapers andidentifyallrelevantpapersintheinvestigateddiscipline;(2)basedonanewexactprocedure,weextendthereview process with multilayernetwork theory; (3) we show that by highlighting thesignificance of identifyingthe emer- gence,growth,reductionandthenetworkofliteratureinadomainordiscipline,scholarscanconductlongitudinaland cross-sectional analysisat thesametime. Toevaluateand showtheadded valueof theSIMILARmethod,we extend theresearchresultsaddressedbyTemplierandParé(2018),whoconductedtheanalysisinthefieldofinformationsys- tems.

Therestofthispaperisorganizedasfollows.Section2providesthetheoreticalbackgroundofthestudy.Section3 introducestheproposedmethod.Section4presentsquantitativeresultsthroughanexampleintheﬁeldofinformation systemsresearch.Section5concludesthepaper.

2. Backgroundofthestudy

Theliteraturereviewprocessisanimportantpartofanyscientificpaperorresearch.Itisvalidtosaythatanappropriate andeffectiveliteraturereviewisnecessarytoobtainupdatedinformationaboutthelatestscientificresultsandtounderstand andidentifytheexistinggapswithintheselectedresearcharea(Hunteretal.,1982).Itisimportanttointroducehowour proposedmethodfitsintothetypologyoftheliteraturereviews,howitsupportstheproposedliteraturereviewprocesses andhowitcontributestotheexistingcitationnetwork-basedapproaches.Therefore,wediscusstherelatedworksinthree categories:(1)thetypologyofliteraturereviews,(2)theliteraturereviewprocessandguidelines,and(3)citationnetworks.

(3)

Z.T.Kosztyán,T.CsizmadiaandA.I.Katona JournalofInformetrics15(2021)101111 2.1. Typologyofliteraturereviews

Thetypologyofliteraturereviewshasbeenconstructedandexaminedbyseveralscholars.Furthermore,severalliterature reviewtypologieshavebeenconstructedbasedondifferentaspects(Templier&Paré,2015).Inthissubsection,weintroduce thefrequentlyappliedtypologies.

Cooper(1985)categorizedliteraturereviewsbasedonthefocus,goal,perspective,coverage,organization,andaudience ofthepapers.Cook,Mulrow,andHaynes(1997)distinguishedbetweennarrativeandsystematicliteraturereviews.The structureofsuchreviewswaslaterreﬁned,andsystematicreviewswerefurtherdistinguishedintoqualitativesystematic literaturereviewsandquantitativesystematicliteraturereviews(ormeta-analyses),asdescribedbyGreen,Johnson,and Adams(2006),whoalsodiscussedthedifferentsubcategoriesofnarrativereviews.AnothertypologywasusedbyCronin etal.(2008),wherenotonlymeta-analysisbutalsometasynthesiswasdescribedasanonstatisticalmethodwiththeaim ofintegratingtheoutcomeofmultiplequalitativeliteraturereviews.

Theaforementionedtypologiesweredevelopedinthemedicalresearcharea;however,thetypologyofliteraturereviews isalsoafrequentlydiscussedtopicintheresearchﬁeldofinformationsystems(Bandaraetal.,2011;Templier&Paré,2018).

Rowe(2014)classiﬁedtheliteraturereviewsacrossfourdimensions:goalwithrespecttotheory,breadth,systematicity, andargumentativestrategy.Paréetal.(2015)extendedthisstructureandprovidedamoredetailedtypologyincludingnine reviewtypes(seeashortdescriptionofeachtypeinTable1).Later,thetwoschemeswerenested,andacombinedtypology wasdeveloped(Templier&Paré,2018).

Toshowthecontributionofourmethodtothedifferentliteraturereviewtypes,wewillusethecombinedtypologyused byTemplierandParé(2018)becauseitisnewlydevelopedandduetoitscombinationoftwotypologies,itcoversmultiple aspects.

2.2. Literaturereviewprocessandguidelines

Notonlytypologiesbutalsoseveralguidelineshavebeenprovidedtosupportscholarsintheliteraturereviewcon- structionprocess.In1982,Cooperdevelopedscientiﬁcguidelinesforconductingintegrativeresearchreviews.Hismodel includedﬁvesteps:(1)problemformulation,(2)datacollection,(3)dataevaluation,(4)analysisandinterpretation,and(5) publicpresentation(Cooper,1982).Althoughthisresearchfocusedonthereviewprocessofpsychologicalsciences,strong literaturereviewmethodologiesweredevelopedlaterinotherdisciplinesaswell(Templier&Paré,2015),suchashealth sciences(seeAveyard,2014;Hewitt-Taylor,2017;Higgins&Green,2011;Liberatietal.,2009),softwareengineering(see Brereton,Kitchenham,Budgen,Turner,&Khalil,2007;Kitchenham&Charters,2007;Petersenetal.,2008),management sciences(Durach,Wieland,&Kembro,2014;Rousseauetal.,2008)andinformationsystems(Okoli,2015;Okoli&Schabram, 2010).

In2007,KitchenhamandCharters(2007)concludedthatthreemainphasesneedtobeincludedtoprovideaneffective literaturereview:(1)planning,(2)carryingoutand(3)reporting.Theﬁrstphasecontainstheformulationoftheresearch questionanddesignofthereviewmethodology.Thecarryingoutphaseincludesseveraltasks,suchascollecting,validating, andevaluatingpapersandanalyzingresults.Thereportingphasereferstothewritingoftheliteraturereviewbasedonthe outcomeoftheprevioussteps(Kitchenham&Charters,2007;Templier&Paré,2015).

Thefundamentalframeworkwaslaterreﬁnedbyextendingittoasix-stepprocessasfollows:(1)formulatingtheproblem, (2) searchingforliterature,(3)screeningforinclusion, (4)accessingquality,(5)extractingdataand (6)analyzingand synthesizingdata(Templier&Paré,2015).

Thispaperproposesacitationnetwork-basedsystematicliteraturereviewprocessthatallowstheresearchertoiteratively refinethepredefinedcategoriesfortheexaminedpapersandidentifyallrelevantstudiesintheinvestigatedresearchfield.

2.3. Citationnetworks

TheanalysisofcitationnetworkshasitsoriginsintheworkofGarﬁeld,Sher,andTorpie(1964),whoshowedthata strongpatterncanbeobservedbetweenhistoricaldescriptioneventsandcitationaldatarelatedtotheevents.Later,the analysisopportunitiesbycitationnetworkanalysiswereextendedthroughtheresearchworkofHummonandDoreian.

Theseauthorsproposedgraphindicesthathelptoidentifythemostimportantsubgraphofthecitationnetwork(Hummon

&Dereian,1989;Hummon&Doreian,1990;Hummon,Doreian,&Freeman,1990).In2003,Batagelj(2003)proposedan improvementregardingtheaforementionedindices,whichallowedresearcherstoapplythemtolargecitationnetworks.

Inthefollowingyears,theidentificationofthemostinfluencingarticleswithinthecitationnetworkbecameanoutstanding researcharea(seeBergsma,Mandryk,&McCalla,2014;Davletov,Aydin,&Cakmak,2014;delaPena,2011;Hong&Do, 2018;Huangetal.,2018;Pichardo-Corpus,Contreras,&delaPe ña,2019;Su,Sun,Xuan,&Shi,2015).Thevisualizationand analysisofcitationnetworksarealsosupportedbyseveralsolutions,suchasVOSviewer(Leydesdorff,Carley,&Rafols,2013;

Leydesdorff&Rafols,2012),HistCite(Garﬁeld,2009;Lucio-Arias&Leydesdorff,2008)orCiteNetExplorer(Maheswaran, 2019).Nevertheless,thesesolutionsdonotallowusingmultilayernetworks.Theanalysisofresearchtrendswithcitation networksalsoreceivedoutstandingattention;however,theproposedmethodsareeitherlongitudinalonlyorcompare thecross-sectionalresultsoftwoperiods(e.g.,Asatani,Mori,Ochi,&Sakata,2018;Benckendorff,1994;Dawson,Gaˇsevi ´c, Siemens,&Joksimovic,2014;Ho,Saw,Lu,&Liu,2014).Inthispaper,wedevelopacitationnetwork-basedliteraturereview

3

(4)

Table1

Descriptionoftheliteraturereviewtypes.

Type Description Example

Narrative Narrativereviewsattempttoidentifywhathasbeenwrittenona subjectortopic.Theysummarizetheextantliteratureandprovide acomprehensivereportonthecurrentstateofknowledgeonthe topicofinterest.Theyusuallyselectivelyfocusonliteratureand evidencethatarereadilyavailabletoresearchers.

Aloini,Dulmin,andMininno(2007)

Descriptive Descriptivereviewsinvestigatematuretopicsandaimat determiningtheextenttowhichabodyofempiricalstudies revealsanyinterpretablepatternsortrendsoveragivenperiodof time.Theycollectandanalyzenumericdatathatreﬂectthe frequencyofthetopics,authorsormethodsfoundintheextant literature.Structuredsearchmethodsaretypicallyusedtoforma representativesampleofalargergroupofpublishedworksthat arerelatedtoaparticularareaofinvestigation.

Avison,Dwivedi,Fitzgerald,andPowell(2008)

Scoping Scopingreviewsareprimarilyconcernedwithemergenttopics andaimatassessingthesizeandscopeoftheavailableliterature andinformingresearchersaboutanewarea.Theyusuallyaddress abroadspectrumoftopics,wheremanydifferentstudydesigns mightbeapplicable.Scopingreviewsusuallyconcludewith potentialimplicationsforpracticeandresearch.

Archer,Fevrier-Thomas,Lokker,McKibbon, andStraus(2011)

Critical Criticalreviewsaimtocriticallyanalyzetheextantliteratureona broadtopictorevealweaknesses,contradictions,controversies,or inconsistencies.Theyattempttotakeareﬂectiveaccountofthe researchthathasbeendoneinaparticularareaofinterestand assessitscredibilityusingappraisalinstrumentsorothermethods.

Criticalreviewsareeitherselectiveorrepresentativeandrarely involveacomprehensivesearchofalloftherelevantliterature.

Schryen(2013)

Meta-analysis Meta-analysesusespeciﬁcdataextractiontechniquesand statisticalmethodstosummarizeandsynthesizepriorempirical resultsonagiventopic.Meta-analysesprovidepreciseestimates ofrelationsbetweenvariablesbecausetheyadjustforthesample sizeandreliabilityofmeasures.Theyareconsideredtobea powerfulmethodofsynthesisthatallowsresearcherstodraw meaningfulinferencesbysettlingexistingcontroversiesthatarise fromconﬂictingempiricalstudies.

OrtizdeGuinea,Webster,andStaples(2012)

Qualitativesystematic Qualitativesystematicreviewsusenarrativemethodstobring togetherthequantitativefindingsofagroupofheterogeneous studies.Theyemploythetypicalsystematicreviewprocessbutuse narrativeandmoresubjectivemethodstobringtogetherthe findingsoftheincludedstudies.Thedefiningelementof qualitativesystematicreviewsistheadoptionofatextual approachintheprocessofanalysisandsynthesis.

Petter,DeLone,andMcLean(2008)

Umbrella Umbrellareviewsaimatsynthesizingtheﬁndingsofprior reviews.Theyintegraterelevantevidencefrommultiple systematicreviewsintooneaccessibleandusabledocumentto addressanarrowresearchquestion.Umbrellareviewsappraise themethodologicalrigorandqualityofevidenceofsystematic reviewsbymeansofexplicitcriteria.

Mbemba,Gagnon,Paré,andCôté(2013)

Theorydevelopment Theoreticalreviewsdrawonexistingconceptualandempirical studiestoprovideacontextforidentifying,describing,and transformingintoahigherorderoftheoreticalstructureand variousconcepts,constructsorrelationships.Theybringtogether diversestreamsofworkandusevariousstructuredapproaches suchasclassiﬁcationsystems,taxonomiesandframeworksto organizepriorresearcheffectively,examinetheir

interrelationships,anddiscoverpatternsorcommonalitiesthat willfacilitatethedevelopmentofnewtheories.

vonKrogh,Haeﬂiger,Spaeth,andWallin(2012)

Realist Realistreviewsaretheory-driveninterpretativereviewsthat inform,enhance,extendoralternativelysupplementconventional systematicreviewsbymakingsenseofheterogeneousevidenceof complexinterventionsappliedindiversecontexts.Theyusually startbyarticulatinglikelyunderlyingmechanismsandthen scrutinizeavailableevidencetodeterminewhetherandwhere thesemechanismsareapplicable.

Wong,Greenhalgh,andPawson(2010)

(5)

Z.T.Kosztyán,T.CsizmadiaandA.I.Katona JournalofInformetrics15(2021)101111 methodthatcanbeextendedtoamultilayerstructureinwhichlongitudinalandcross-sectionalanalysiscanbeconducted atthesametime.

3. Methods

InSection3.1,theappliedtermsanddeﬁnitionsarepresented.InSection3.2,thestepsoftheproposedmethodare shown.Finally,inSection3.3,theoperationalizationoftheproposedmethodisintroduced.

3.1. Termsanddeﬁnitions

Graph:GraphisatupleG=(V,E),whereVisasetofnodes(V={v1,v2,...,vn})andEissetofedgeswheree_ijrepresents theedgebetweennodesviandvj(i,j∈{1,...,n},nisthenumberofnodes).Differentmetricscanbeusedtodeterminethe

“importance”ofthepapersortocharacterizeeachcategory(inotherwords,agroupofspeciﬁcstudies)acrossthecitation network.Forthispurpose,weusedensity,numberofcomponentsandmeanpathlength.

Graphdensity:Densitycanbecalculatedasfollows:

D= a

n(n−1)/2 (1)

whereaisthenumberofactualedgesinagraph(orsubgraph),andn(n−1)/2denotesthepossiblenumberofedgesamong thennodesofthegraph.Weusegraphdensitytomeasurehowthestudiesinaparticulararearelyoneachother’soutcomes (citeeachother).

Connectedgraph:Agraphisconnectedifithasapathbetweeneverypairofnodes.

Numberofcomponents:IfGisagraphwithHsubgraph,thenHisacomponentofGif(1)itisconnectedand(2)H isnotconnectedtoanyconnectedsubgraphofG(Chartrand,1977).Thenumberofcomponentsrevealsinformationabout isolatedresearchwithinaspeciﬁcpartofaresearchﬁeld.

Meanpathlength:Meanpathlength(lG)canbecalculatedasfollows:

l_G= 1 n(n−1)

i=/j

d(vi,vj) (2)

where d(vi,vj)istheshortestpathbetweennodes viand vj. Thismetricis usedtocharacterizethelength(speed)of informationspreadingacrossthespeciﬁcresearchsubarea(subgraph).

Multilayernetwork:Theproposedmethodbasedonamultilayernetworkanalysis,whereamultilayernetworkisapair M=(G,C)G={G˛=(V˛,E˛),˛ ∈{1,...,M}}isafamilyof(directedorundirected,weightedorunweighted)graphs (calledlayersofM),V˛isthesetofnodes(vertices),E˛isthesetofedges(arcs)ofgraphG˛,andthefollowing:

C={E_˛,ˇ⊆V˛×V_ˇ,˛,ˇ ∈{1,...,M},˛=/ˇ} (3) isthesetofinterconnectionsbetweennodesofdifferentlayersG˛andG_ˇwith˛ =/ ˇ.

Reducibilityofmultilayernetwork:Thenumberoflayersinthemultilayercitationnetworkisanimportantquestion.

Todeterminetheoptimalnumberoflayers,wefollowtheapproachdevelopedbyDeDomenico,Nicosia,Arenas,andLatora (2015),whousedtheJensen–Shanondivergencetomeasurethepairwisesimilaritybetweenthelayers.TheJensen–Shanon divergence(D_JS)isamodiﬁcationoftheKullback–Lieblerdivergence(D_KL).Letandbetwodensitymatriceswith respecttotwodifferentlayers(G˛andG_ˇ);then,D_KLcanbedescribedasfollows:

D_KL()=Tr[(log₂(−log₂())] (4)

Trdenotesthetraceofthesquarematrix,whichisthesumofallelementsinthemaindiagonalofthematrix.SinceD_KLis notsymmetric(D_KL()=/D_KL()),D_JSisusedinsteadasfollows:

D_JS()= 1

2D_KL()+1

2D_KL() (5)

Theresultscanbeillustratedthroughadendrogramindicatinghowtheinitialstructurecanbeaggregated.Reducibility analysisisveryimportantbecauseitdeterminestheunitofanalysis,e.g.,theoptimalstepsizeifcitationnetworklayersare constructedbasedontime.

3.2. Stepsoftheproposedmethod

Theproposedmethodconsistsofthefollowingfourfundamentalsteps:

1.Metastructuresearch

2.Explorationofcitationnetwork 3.Relevanceexamination 4.Classiﬁcation

5

(6)

Fig.1. Metastructureexample.

Fig.2.Metastructurewithconnectedpapers.

Table2providesanoverviewofthestepsoftheproposedapproach.

Step1:Metastructuresearch.Inthefirststep,apublishedandrelevantliteraturereviewmustbeselectedforuseas aninitialpointforthecurrentliteraturereviewprocess.Theselectedliteraturereviewmustcontaintheclassificationof reviewedarticlesbecauseitsclassificationruleswillbeusedandfurtherexpandedorimprovedintheremainingstepsif necessary.Inthispaper,werefertotherulesofclassificationas“metastructure”.Thismetastructurewillbeexpandedand validatediterativelyduringtheapplicationoftheproposedmethod.Nodesrepresentdifferentresearchareaswithinthe selecteddiscipline,andedgesrepresentthestructureamongthem(Figs.1and2).

Classiﬁedpapersbasedontheselectedliteraturereviewneedtobeconnectedtothebuiltmetastructure.Basically,this stepsummarizesandvisualizestheresultsgivenbytheinitialliteraturereview.

Step2:Explorationofthecitationnetwork.Thesecondstepistheexplorationofthecitationnetworkaccordingtothe derivedmetastructureanditsclassiﬁedpapers.Alistofcitingpapersmustbecollectedforalltheinitiallyclassiﬁedpapers (fromstep1)withoutconsiderationofrelevance,sinceirrelevantpapers(nodes)willberemovedaspartofafurtherstep.

Notonlycollectionbutalsoconnectiontotheappropriatenodesisnecessary.Letnode“A”representtheclassiﬁedpaperby theinitialliteraturereviewandnode“B”representoneelementofthelistofcitingarticlesregardingnode(paper)“A”.In thiscase,adirectededgeneedstobedrawnpointingfromnode“B”to“A”,indicatingthatpaper“A”wascitedbypaper“B”.

Thisstepmustbeconductedforallthepapers(nodes),includingthoseintheinitiallyselectedliteraturereview(Fig.3).

Step3:Relevanceexamination.Inthisstep,therelevanceoftheexploredcitingpapersisexamined,andirrelevant papersmustberemovedfromtheconstructedcitationnetwork(givenbystep2).Tojudgetherelevanceofthecollected papers,thetitles,abstracts,keywordsandcontentcanbeused.Althoughthisstepistimeintensive,thereisanopportunity forfurtherresearchtobuildmachinelearning-basedclassiﬁerstosupportthistaskanddecidewhethertheexaminedpaper isrelevantfromtheperspectiveoftheliteraturereview(Fig.4).

Step4:Classification.Thelaststepisclassification,wheretheconstructedcitationnetworkneedstoberevised.Each connectedpapermustbeanalyzedintermsofitsrelationtothemetastructure.Iftheithpaper’stopicfitstothearea describedbythepathofthemetastructure,thenitslocationcanbekept.Otherwise,ifitstopicisanewareawithinthe analyzeddiscipline,thepathwithinthemetastructure(thatincludesthecurrentpaper)needstoberevised.Thefollowing actionscanbemade:

•Createanewresearcharea

(7)

Z.T.Kosztyán,T.CsizmadiaandA.I.KatonaJournalofInformetrics15(2021)101111

Table2

Stepsoftheproposedapproach.

Nr Step Description Assumption Outcome Stoppagecriteria Opportunitiesto

support 1 Findingarelevant

literaturereview.

Apublishedandsuitableliteraturereview mustbefound,whichalsodescribesthe typology/classiﬁcationofapproachesinthe selectedresearchtopic.Ifmorereviewscanbe found,theresearchercandecidetousethe(1) unionor(2)thesetofmultipleliterature reviews,or(3)themostsuitable/citedreview canbeselected.

Atleastoneliterature reviewexistsinthe studiedresearchﬁeld thatdescribesthe typologyoftheofthe selectedresearchtopic.

1.Ametastructurethat describesthetypology oftheselectedresearch topic.2.Asetof classiﬁedpapersbased ontheselected metastructure.

1.Anappropriateliterature reviewwasfoundwithsuitable classiﬁcationrules2.An appropriateliteraturereview wasselectedfromseveral candidates.

GoogleScholar keywordsearch.

Speciﬁcterms:

literaturereview survey.

2 Explorationofthe citationnetwork.

Thecitationlistmustbeextractedforallthe categorizedpapersgivenbytheliterature reviewdeﬁnedatstep1.Inthisway,the numberofcategorizedpaperswillbe increasedsigniﬁcantly.Thelistofcitingand citedarticlesmustbestoredinordertoensure alltherelevantpapersareincludedinthis expandeddataset.

Citationinformation mustbeknowntobe relatedtothepapers.

Anexploredcitation networkbasedonthe selectedmetastructure.

1.Onlynewapplicationsof existingsolutionsbutno furtherapproachescanbe foundintheexaminedﬁeld.2.

Furtherarticleshavealow citationnumber(low relevance).

Existingcitation connectionscanbe foundautomatically withscripts/macros.

3 Relevancechecking. Byaddingallthecitedandcitingarticles,the networkisalsoincreasedwithbytheaddition ofpapersthatarenotrelevantaccordingtothe selectedresearchtopic.Thesearticlesmustbe identiﬁedandremovedfromthedataset.

Title,abstract, keywordsofthe exploredpapers.

Acleanedcitation networkbasedonthe selectedmetastructure.

Alltheexploredandsaved papersareexaminedfromthe perspectiveofrelevance.

1.Filteringby keywords2.Filtering bybigrams/trigrams usingtheabstract.

4 Classiﬁcation. Ifthecitationnetworkiscleanedproperly(i.e., allthenonrelevantpapersareremoved),then thenewlyexploredpapersmustbeexamined fromtheclassiﬁcationperspective.Ifpapers canbefoundinwhichnewapproacheswere developed,themetastructureneedstobe adjusted,andifnecessary,new

categories/subcategoriesneedtobeaddedto theinitialmetastructure.

Acleanedandexplored citationnetworkthat includestheinitial classiﬁcationofthe exploredpapers.

Anupdated,relevant citationnetworkbased onthereﬁnementof theinitial metastructure.

Alltherelevantpapersinthe datasetwereexamined regardingthecorrectnessof theclassiﬁcation.

7

(8)

Fig.3.Explorationofcitationnetwork.

•Mergemultipleresearchareas

•Splittheexistingresearchareaintomultipleareas

Thisphaseendswhenallthepapersarerevised(notethatonlyrelevantpapersareincludedatthisstepbecausenonrel- evantnodeswereremovedastheoutcomeofstep3).Thisstepensuresthattheresearcherﬁndsthoseresearchareasthat cametoexistenceaftertheoriginalmetastructurewascreatedbytheselectedliteraturereview(Fig.5).

3.3. Evaluationprocess

Todemonstratetheapplicabilityoftheproposedmethod,weapplythefollowingstepsduringtheevaluation:

I.DataCollection

(a)Noniterativephase:Searchforandselectionofanexistingsystematicliteraturereview.

(b)Iterativephases:Applicationoftheproposedmethodforﬁndingadditionalpapersandtransformingthemintothe networkstructure.

II.DataAnalysis

(a)Single-layernetworkanalysis (b)Multilayeranalysis

-Reducibilityanalysis

-Analysisacrossdifferentlayers

Tofacilitatethereproducibilityoftheproposedmethod,wegiveadetaileddescriptionoftheevaluationprocess.

I.a.DataCollection-Noniterativephase.Thisphasecoversstep1:metastructuresearchdescribedinSection3.2.The examplefocusesonthecategorizationofexistingliteraturereviewtypes,inwhichweperformakeywordsearchinGoogle Scholartofindthemostappropriateone.TheliteraturereviewconductedinthefieldofinformationsystemsbyTemplier andParé(2018)wasthestartingpointoftheanalysis;weusedtheirtypologyasametastructuresinceitisacomprehensive andwell-presentedoverviewofthefield.

(9)

Z.T.Kosztyán,T.CsizmadiaandA.I.Katona JournalofInformetrics15(2021)101111

Fig.4.Relevanceexamination.

I.b.DataCollection–Iterativephases.Thisphase coverstheiterativesteps,i.e.,step2:explorationofthe citation network,step3:relevanceexaminationandstep4:classification,introducedinSection3.2.First,weexploredthecitation dataamongthepaperscategorizedbytheresultsofTemplierandParé(2018).Then,thiscitationnetworkwasfurther extendedbyadditionalpapersfoundbyakeywordsearchinGoogleScholar.Arelevanceexaminationandclassification wereperformedbytworeviewers.Toexaminetherelevance,thetitle,abstractandkeywordsofthepaperswerereviewed (theseinformationwereobtainedfromthegivenjournal’shomepage).Fortheclassification,thedefinitionsofeachcategory providedbyTemplierandParé(2018)wereconsidered.Afterassigningtheappropriatecategory,thecitationsbetweenthe newlyfoundpaperandthealreadyincludedonesweredeterminedbyapairwisecomparisonofthereferencelists.The nextiterationincludesoneofthecitationreferencesofthenewlyaddedpaper.Thesestepswereiterateduntilnofurther relevantpaperwasfound.

II.DataAnalysis.Toconductthecitationnetworkanalysis,additionaldata,suchastheyearlynumberofcitationsand journalH5indices(representingthejournalquality),werecollectedforeachpaperinthereview.Thenumberofcitations

9

(10)

Fig.5. Classiﬁcation.

Fig.6.Single-layercitationnetworksonliteraturereviews.

andH5indiceswereextractedusingGoogleScholar’sfeatures.Astheinput,twotables,i.e.,onelistingthenodes(papers) andonelistingtheedges(citations),werepopulatedinMSExcel.Single-layernetworkanalysiswasconductedintheR programusingthe“igraph”package(Csárdi&Nepusz,2006).Formultilayernetworkanalysis,weusedtheMuxViz(De Domenico,Porter,&Arenas,2014)open-accessplatform.Tosupportreproducibility,allmultilayernetworkswereattached tothepaper.

Section4introducestheresearchresultsobtainedbyfollowingthestructuredescribedabove.

4. Results

Theaimofthissectionisthreefold.First,thestudyshowshowtheproposediterativemethodcanﬁndnewandrelevant papersbasedonthetypesofliteraturereview.Second,withtheproposedgeneralmultilayernetworkapproach,theevolution ofliteraturereviewsisanalyzed.Third,theimpactsofthedifferentcategoriesofliteraturereviewsarealsoanalyzedbytheir existenceinthehigherprestigiouslayers.

(11)

Table3

Descriptivepropertiesofthesubgraphs.

Subgraph n Edges D C MPLG Allcitations

Critical 19 1 0.003 18 1.00 9326

Descriptive 33 14 0.013 20 1.13 8397

Meta-analysis 26 24 0.037 11 1.32 4705

Narrative 35 15 0.013 20 1.12 20,105

Qualitativesystematic 11 15 0.136 5 1.06 4205

Realist 8 1 0.013 7 1.00 910

Scoping 15 3 0.014 12 1.00 3619

Theorydevelopment 65 97 0.023 5 1.28 39,324

Umbrella 12 7 0.053 5 1.36 715

Fig.7.Cumulativeratioofpublishedarticlesbyyear.

4.1. Datasourceandcitationnetworkconstruction

TheinitialdatasetcollectedbyTemplierandParé(2018)included142papersacrosssevenliteraturereviewtypes(there werenopapersincludedintherealistorumbrellareviewcategories).Afteraniterativedatacollectionbasedontheproposed method,thedatasetwasextendedwiththeadditionofpapersforatotalof224;however,this studyisnotlimitedto thespeciﬁcjournalsspeciﬁedbyTemplierandParé(2018).AsSection3.3describes,notonlyadditionalpapersbutalso supplementarycitationdatawerecollected:(1)citationsamongpapersinthedatasetwererevealed,and(2)theannual numberofcitations(numberofcitationsbyarticleswithinoroutsidethecollecteddataset)wasextractedforeacharticle.

4.2. Single-layernetworkanalysis

Fig.6showstheresultsoftheexploration.Thesizeofthenodesisrelatedtothenumberofcitations.Blacknodes(14) representthemetastructure,thecolorednodes(142)representtheoriginallycollectedandclassifiedpapers,andthewhite nodes(82)aretheadditionallyexploredpapers.Notonlywastheoriginallistofthepapersextended,butevenempty categories,suchasrealistandumbrellareviews,werefilledbyapplyingtheSIMILARmethod.Nevertheless,wecanconclude thatneitherTemplierandParé(2018)northeSIMILARmethodfoundanewresearchfieldasanadditionalliteraturereview category.

TheSIMILARmethodproveditselftobeacomprehensivemethodinfindingalltherelevantliterature.TemplierandParé (2018)limitedtheirresearchtoonlyeighthigh-qualityjournals.TheSIMILARmethodwasabletofindfurtherrelevanthighly citedreviewpaperswithinandoutsidetheinitiallyselectedjournalsofTemplierandParé(2018).Therefore,theextended datasetcanincludenewtoppapersevenfromthefirstquartile,whichisalsoindicatedbythesizeofthewhitenodesin Fig.6.Whenconductingacomprehensiveliteraturereview,thispapersuggeststheconsiderationofcitationnumbersduring relevancecheckinginadditiontothepreliminaryspecifiedsetofjournals.

TheSIMILARmethodalsorevealssomeimportantfeaturesofthedifferentliteraturereviewtypes.First,thecitation networkdepictsthatthenumberofedges(i.e.,thenumberofcitations)isthehighestinthetheorydevelopmentcategory (97,seeTable3),indicatingthatthiscategoryisthemostpopular.Second,thedensityofthesubgraphsprovidesadditional

11

(12)

Fig.8. Reducibilityanalysisofthemultilayernetwork.

informationoncategories.Thesubgraphsinthemaincategoryof“testingtheory”showhighdensity(seeFigure6),indicat- ingthatdespitethelowernumberofpapers(nodes),scholarsrelyonpapersfromthesamecategorymorestronglyinthe caseofqualitativesystematic,meta-analysisandumbrellareviews(seeTable3).Incontrast,thedensityofcriticalreviews isthelowest(0.003),whichindicatesthatcriticalreviewsareveryrarelycitedbyothercriticalreviews.Third,theanalysis ofthecomponentsshowsthefragmentationofthesubgraphs.Comparedtothenumberofpapersincriticalreviews(19), thenumberofcomponentsisoutstandinglyhigh(18),whichmeansthatthiscategoryisveryfragmented.Theorydevel- opment,however,hasonly5componentsbutalsothehighestnumberofpapers(65),indicatingthattheresearchresults areseparatedintoalownumberofsegments.Fourth,thehighvalueofthemeanpathlengthshowsthatumbrellareview paperssequentiallyutilizetheirresultsovertime.Finally,consideringthenumberofcitations,weconcludethattheory developmentandnarrativereviewsarethemostcitedtypesofall.

4.3. Multilayernetworkanalysis

Inadditiontothesingle-layernetworkanalysis,theproposedmultilayernetworkenablestheanalysisofthedynamics ofcitationbehavior.Itisnecessarytonotethatthisdynamiccanrepresenttimeorotherfeatures,suchasjournalquality, location,andinstitute.Thissectionfocusesonpublicationyearandjournalqualityfeaturesasthebasisofthemultilayer structure.

Multilayernetworkanalysisbasedonpublicationyear.First,weanalyzethenetworkstructurebasedontheyearofpublica- tion.Astheinitialstepofanydynamicmultilayernetworkanalysis,theunitofanalysisneedstobespeciﬁed.Toovercomethe

(13)

Fig.9. Multiplexmultilayernetwork-basedonyearofpublication.

Fig.10.Averagecitationnumberbyyear.

distortionofthefrequenciesbycategory,wecalculatethecumulativeratioofpapersineachpublicationyearperliterature reviewtype(seeFig.7).

Thecumulativeratioofpaperscanbedescribedwellbylineartrends,andshiftscannotbeobserved,indicatingthe homogeneousbehaviorofpublicationactivityovertime.Thissuggeststhatmultilayeranalysisshouldbeconductedona yearlybasis.Weusetheseresultsasareferencetoanalyzewhetherthemultilayerreducibilityanalysisreturnsaresimilar;

however,itisamoresophisticatedoutcome(seeFig.8).

Theresultgivenbythereducibilityanalysisconﬁrmingtheinitialassumptionshowsthateverysinglelayershouldbe treatedasastandalonelayerexceptlayers“2002”and“2003”.Inthiscase,however,merginglayers“2002”and“2003”does notgrantanyremarkablesimpliﬁcationrelatedtotheanalysis.Thus,inthispaper,weconductmultilayernetworkanalyses onayearlybasis.Fig.9showsthemultilayercitationnetworkinwhicheachlayerrepresentsonepublicationyear.

Similartothesingle-layeranalysis(seeFig.6),inthistypeofanalysis,blacknodesrepresentthemetastructure,and colorednodesdenotethecollectedpapersinaparticularpublicationyear(layer).Thenodesizerepresentsallthenumber ofcitationsreceivedinthesameyear(layer).Theintralayeredgesshowthecitationsamongthearticlespublishedinthe sameyear,whiletheinterlayerconnectionsrepresentthecitationsfromdifferentyears.Themultilayercitationnetwork

13

(14)

Fig.11.Citationbehaviorofreviewpapersinthetheorydevelopmentandmeta-analysiscategories.

isapowerfultoolforexaminingmultiplecharacteristicsatthesametime.Fig.9clearlyindicatesthatnarrativereviews densiﬁedinthetimeperiodof2004–2012,whileumbrellareviewsreceivedincreasedattentionfrom2013.Inaddition,the multilayeranalysisalsoindicatesthatanumberoftheorydevelopmentreviewsarehighlycitedevenintheyearoftheir publication.

Inaddition,Fig.10presentstheaveragenumberofallcitations(citingpapersevenoutsidethecollecteddataset)foreach categoryperyear.Narrativereviewsrapidlyemergeduntil2003andremainedhighlycitedafterwards.Thecitationtrends forallofthecategoriesexceptscopingreviewsbecameﬂatafter2013.Thisanalysisalsohighlightsthattheexaminedreview categoriescanbeclassiﬁedintotwogroupsbasedontheaveragenumberofcitationsin2019.Theuppergroupcontains5 reviewtypes,namely,critical,theorydevelopment,qualitativesystematic,narrativeandscopingreviews,withgreaterthan 40citationsforapaperonaverage.

Fig.11grantsfurtheranalysisopportunitiesbyextendingtheexaminationwithanadditionaldimensionsuchasthe yearofpublication.Thenatureofthecitationsispresentedbysmoothedcontourmapsbasedonfrequencymatrices.The horizontalaxisshowstheyearofthecitingpapers,whiletheverticalaxisdenotestheyearofpublications.Theupper triangleismissingbecausethecitationscomeuponlyafterpublication.Twocategories,namely,theorydevelopmentand meta-analysis,areselectedtocharacterizetheircitingbehavior.

InFig.11aandb,thecontourmapsareonlycalculatedaccordingtothecollecteddataset,whileFig.11canddshowsall thecitationsextractedfromGoogleScholar.Inthecaseofthetheorydevelopmentcategory,consideringcitationsamongthe collectedISliteraturereviews(Fig.11a),themostcitedpaperswerepublishedin2000–2001,andthecitationsmainlycame fromtheperiodof2002–2012.Thispatternisstillvalidifallthecitationsaretakenintoaccount(Fig.11c);however,the emphasisshiftstotheperiodof2010–2019.Inthecaseofmeta-analysisreviews,mostofthecitedpaperswerepublished around2004,andtheyreceivedthemostcitationsafter2012,consideringthecitationsamongthecollectedISliterature reviews(Fig.11c).However,thecitingbehaviorshowsadifferentpatternifallthecitationsaretakenintoaccount.Two peakscanbeobserved,meaningthathigh-qualitymeta-analysispaperswerewrittenﬁrstinthe2002–2006periodand secondinthe2012–2017period(Fig.11d).

Multilayernetworkanalysisbasedonjournalquality.Journalqualityisanotherimportantfeaturethatcanbeusedto constructthemultilayercitationnetworkstructurebecauseitrevealswhichcategoriesarepublishedbyprestigiousjournals.

(15)

Fig.12.RatioofpapersbyH5index(cumulative).

Fig.13. Structuralreducibility.

15

(16)

Fig.14.Multilayernetwork-basedonjournalquality.

Followingthelogicappliedintheyearlyanalysis,thecumulativeratioofpapersiscalculatedbytheH5indextoovercome thedistortionofthefrequenciesbycategory(seeFig.12).

Unlikethatfortheyearlyanalysis,Fig.12showsaremarkableshiftinthecumulativedistributions,suggestingthatlayers couldpossiblybemergedbythemultilayeranalysisrelatedtojournalquality.Tospecifytheunitoftheanalysis,areducibility analysiswasconducted.Fig.13showstheclustereddistancematrixoflayers,suggestingtwolayersformultilayernetwork construction(seeFig.14a).Fig.14bconﬁrmsthatusinganadditionallayerdoesnotindicateanystructuraldifferences.

Theresultsshowthatonlyafewliteraturereviewtypes,suchasumbrella,meta-analysisandtheorydevelopment,are publishedinthehighestprestigiousjournals(seeFig.14a,layer89–176).

5. Conclusion

Asthenumberofpublishedscientificpapershasincreasedexponentiallyoverthepastdecades,conductingeffectiveand efficientliteraturereviewsisessential.ThispaperpresentsanovelmethodcalledSystematicIterativeMultILAyerliterature Review(SIMILAR),whichcanovercomeseveralshortcomingsofexistingmethods.TheproposedSIMILARmethodconsists offourfundamentalsteps:(1)metastructuresearching,(2)explorationofcitationnetwork,(3)relevanceexaminationand (4)classification;thisexactprocedureenablesthecomprehensiveexplorationofagivendisciplinaryfieldorresearcharea.

Themeritsofourmethodincludethefollowing:(1)theSIMILARmethoditerativelybuildsacitationnetworkthatensures theminimizationofinformationlossandacomprehensiveliteraturereview;(2)thecitationnetworkistransformedintoa multilayerstructureallowingthesimultaneouslongitudinalandcross-sectionalanalysisofadisciplinaryﬁeldorresearch area;(3)multilayeranalysisenablesthespreadofmodelinginformationacrossthenetwork;and(4)multilayercitation networksubgraphscanbecharacterizedobjectivelybyusingappropriatenetworkfeaturessuchasdensity,diameterand meanpathlength.

Bydemonstratingtheproposedmethodinthefieldofinformationsystems,weextendedtheresearchresultsaddressed byTemplierandParé(2018)asaninitialdataset.Werevealednewpapersandfilledagapbyfindingliteraturebasedon twomissingreviewtypes;wealsoprovidedadditionalinformationabouttheyearlypublicationactivityandthequalityof reviewarticlesbasedontheHindex.

Finally,choosingthemostappropriate literaturereviewastheinitialstructure canbechallenging,especiallywhen multiplecandidatesareavailable.Tochoosethemostappropriateone,wesuggestconsideringthepublicationyearofthe reviewasadecisivefactor,sinceusinganewercategorizationwillresultinfewerchangesinthemetastructureduring theiterations.Furthermore,ifnosigniﬁcantdifferenceexistsbetweenthepublicationyearsofseveralreviews,wepropose