A Day in Life of Business Intelligence (BI) Architect- part 1

BI Architect most important responsibility is maintaining semantic Layer between Datawarehouse and BI Reports.
There are basically Two Roles of Architect: BI Architect or ETL Architect in data warehousing and BI. (ETL Architect in Future posts).
Semantic Layer Creation
Once data-warehouse is built and BI reports Needs to created. Then requirement gathering phase HLD High level design and LLD Low Level design are made.
Using HLD and LLD BI semantic layer is built in SAP BO its called Universe, in IBM Cognos using framework manager create Framework old version called catalogue, In Micro strategy its called project.
Once this semantic layer is built according to report data SQL requirements.
Note: Using semantic layer saves lot of time in adjustment of changed Business Logic in future change requests.
Real issues Example: Problems in semantic Layer creation like in SAP BO: Read
https://sandyclassic.wordpress.com/2013/09/18/how-to-solve-fan-trap-and-chasm-trap/
Report Development:
Reports are created using objects created by semantic layer.Complex reporting requirement for
1. UI require decision on flavour of reporting Tool like within
There are sets of reporting tool to choose from Like in IBM Cognos choose from Query Studio, Report Studio, Event Studio, Analysis Studio, Metric Studio.
2. Tool modification using SDK features are not enough then need to modify using Java/.net of VC++ API. At html level using AJAX javascript API or integrating with 3rd party API.
3. Report level macros/API for better UI.
4. Most important is data requirement my require Coding procedure at database or consolidations of various databases. Join Excel data with RDBMS and unstructured data using report level features. Data features may be more complex than UI.
5. user/data level security,LDAP integration.
6. Complex Scheduling of reports or bursting of reports may require modification using rarely Shell script or mostly Scheduling tool.
List is endless
Read More:
details of
https://sandyclassic.wordpress.com/2014/01/26/a-day-in-life-of-bi-engineer-part-2/

Integration with Third party and Security

After This BI’s UI has to fixed to reflect customer requirement. There might be integration with other products and seamless integration of users By LDAP. And hence Objects level security, User level security of report data according to User roles.
Like a Manager see report with data The same data may not be visible to clerk when he sees same report. Due filtering of data by user roles using User Level security.

BI over Cloud
setting BI over cloud Read blog.
Cloud Computing relation to Business Intelligence and Datawarehousing

Read :
1. https://sandyclassic.wordpress.com/2013/07/02/data-warehousing-business-intelligence-and-cloud-computing/

2. https://sandyclassic.wordpress.com/2013/06/18/bigdatacloud-business-intelligence-and-analytics/

Cloud Computing and Unstructured Data Analysis Using
Apache Hadoop Hive
Read: 
https://sandyclassic.wordpress.com/2013/10/02/architecture-difference-between-sap-business-objects-and-ibm-cognos/
Also it compares Architecture of 2 Popular BI Tools.

Cloud Data warehouse Architecture:
https://sandyclassic.wordpress.com/2011/10/19/hadoop-its-relation-to-new-architecture-enterprise-datawarehouse/

Future of BI
No one can predict future but these are directions where it moving in BI.
https://sandyclassic.wordpress.com/2012/10/23/future-cloud-will-convergence-bisoaapp-dev-and-security/

A day in life of BI Engineer part 2

Read Part1:
https://sandyclassic.wordpress.com/2014/01/26/a-day-in-life-of-business-intelligence-engineer/
Part 2:
First few days should understand business otherwise cannot create effective reports.
9:00 -10am Meet customer to understands key facts which affect business.
10-12 prepare HLD High level Document containing 10,000 feet view of requirement.
version 1. it may refined later subsequent days.
12-1:30 attend scrum meeting to update status to rest of team. co-ordinate with Team Lead, Architect and project Manager for new activity assignment for new reports.
Usually person handling one domain area of business would be given that domain specific reports as during last report development resource already acquired domain knowledge.
And does not need to learn new domain..otherwise if becoming monotonous and want to move to new area. (like sales domain report for Chip manufactuers may contain demand planning etc…)
1:30-2:00 document the new reports to be worked on today.
2:00-2:30 Lunch
2:30-3:30 Look at LLD and HLD of new reports. find sources if they exist otherwise Semantic layer needs to modified.
3:30-4:00 co-ordinate with other resource reports requirement with Architect to modify semantic layer, and other reporting requirements.
4:00-5:00 Develop\code reports, conditional formatting,set scheduling option, verify data set.
5:00-5:30 Look at old defects rectify issues.(if there is separate team for defect handling then devote time on report development).
5:30-6:00 attend defect management call and present defect resolved pending issue with Testing team.
6:00-6:30 document the work done. And status of work assigned.
6:30-7:30 Look at report pending issues. Code or research work around.
7:30-8:00 report optimisation/research.
8:00=8:30 Dinner return back home.
Ofcourse has to look at bigger picture hence need to see what reports other worked on.
Then Also needed to understand ETL design , design rules/transformations used for the project. try to develop frameworks and generic report/code which can be reused.
Look at integration of these reports to ERP (SAP,peopesoft,oracle apps etc ), CMS (joomla, sharepoint), scheduling options, Cloud enablement, Ajax-fying reports web interfaces using third party library or report SDK, integration to web portals, portal creation for reports.
So these task do take time as and when they arrive.

Enterprise service Bus ESB, BPM orchestration and Process Re-engineering

This is next topic going to write
I covered ESB and BI architecture similarity in last post.
ESB_arch
https://sandyclassic.wordpress.com/2013/10/02/architecture-difference-between-sap-business-objects-and-ibm-cognos/
Every Java web server has its own ESB in market like BEA weblogic’s Aqualogic as shown above.
BPM business Process Re-engineering BPR efforts are on everywhere due to troubled times this where Business Process Managment BPM products comes in where you can dynamically orchestrate business Processes to reflect change quite quickly using BPM products. Each Business Process is basically a web service or bundle of web service.
some like Pega BPM provide Extra feature:
http://www.pega.com/products/business-process-management
then there is XML structured language for BPM called BPEL Like Oracle BPEL which is declarative language for orchestrating Business Processes.

There vendor which have All in one product like Oracle Weblogic BPM, BPEL, ESB.
Aqualogic ESB is more popular

 

Approach to Best collaboration Management system

Collaboration tools integrated offering (course grain integration using ) integration tools like TIBCO, Oracle BPEL, : Components to be integrated:
1. Content management system CMS  (SharePoint, Joomla, drupal) and
2. Document Management system like (liferay, Document-um, IBM file-net) can be integrated using flexible integration tools.

3. Communication platform like Windows Communication Foundation ,IBM lotus notes integrated with mail client and Social network like Facebook using Facebook API, LinkedIn API, twitter API ,skype API to direct plugin as well as data Analysis of Social networking platform unstructured data captured of the collaboration for the project discussion.
soft-phone using Skype offering recording conversation facility for later use.

https://sandyclassic.wordpress.com/2013/06/19/how-to-do-social-media-analysis/

Oracle Web centre:
https://sandyclassic.wordpress.com/2011/11/04/new-social-computing-war-oracle-web-centre/
4. Integrated Project specific Wikki/Sharepoint/other CMS pages integrated with PMO site Artefacts, Enterprise Architecture Artefacts.
5. seamless integration to Enterprise Search using Endeca or Microsoft FAST for discovery of document, information, answers from indexed,tagged repository of data.
6. Structured and Unstructured data : hosted on Hadoop clusters using Map-reduce algorithm to Analyse data, consolidate data using Hadoop Hive, HBase and mining to discover hidden information using data mining library in Mahout for unstructured data.
Structured data kept in RDBMS clusters like RAC rapid application clusters.
https://sandyclassic.wordpress.com/2011/10/19/hadoop-its-relation-to-new-architecture-enterprise-datawarehouse/


https://sandyclassic.wordpress.com/2013/07/02/data-warehousing-business-intelligence-and-cloud-computing/
7. Integrated with Domain specific Enterprise resource planning ERP packages the communication, collaboration,Discovery, Search layer.
8. All integrated with mesh up architecture providing real-time information maps of resource located and information of nearest help.
9. messaging and communication layer integrated with all on-line company software.
10.Process Orchestration and integration Using Business Process Management tool BPM tool, PEGA BPM, Jboss BPM , windows workflow foundation depending landscape used.
11. Private cloud integration using Oracle cloud , Microsoft Azure, Eucalyptus, open Nebula integrated with web API other web platform landscape.
https://sandyclassic.wordpress.com/2011/10/20/infrastructure-as-service-iaas-offerings-and-tools-in-market-trends/
12. Integrated BI system with real time information access by tools like TIBCO spotfire which can analyse real time data flowing between integrated systems.
Data centre API and virtualisation plaform can also throw in data for analysis to hadoop cluster.
External links for reference: http://www.sap.com/index.epx
http://www.oracle.com, http://www.tibco.com/,http://spotfire.tibco.com/,
http://scn.sap.com/thread/1228659
S
AP XI: http://help.sap.com/saphelp_nw04/helpdata/en/9b/821140d72dc442e10000000a1550b0/content.htm

Oracle Web centre: http://www.oracle.com/technetwork/middleware/webcenter/suite/overview/index.html

CMS: http://www.joomla.org/,http://www.liferay.com/http://www-03.ibm.com/software/products/us/en/filecontmana/
Hadoop: http://hadoop.apache.org/

Map reduce: http://hadoop.apache.org/docs/stable/mapred_tutorial.html
f
acebook API: https://developers.facebook.com/docs/reference/apis/
L
inkedin API: http://developer.linkedin.com/apis
T
witter API: https://dev.twitter.com/

Enterprise Architecture TOGAF,ITIL,Zachman,eTom,NGOSS

A list of ALL ENTERPRISE FRAMEWORKS…

– Business Frameworks (e.g. BMM, Six Markets, Porters 5 Forces, McKinsey 7S, etc)
– Architecture Frameworks (e.g. PEAF, MODAF, TOGAF, TEAF, IAF, etc)
– Programming Frameworks (e.g. .NET, J2EE, etc)
– Project Management Frameworks (e.g. PRINCE2, MSP, etc)
– Service Management Frameworks (e.g. ITIL, etc)
– Industry Operations Frameworks (e.g. eTOM, Pragmatic Marketing, etc)
– Any other type of framework in any other domain so long as it is related to an ENTERPRISE.

There are various field of Architecture floating around in IT industry.Specially in indian IT industry where things are still taking shape.HR department is india in most forgotten department it still need to learn more about Labour laws and nothing more..Top institution in india for HR which was essentially focus on labour laws and nothing more.Other subjects are taught not with great emphassis..Problem that is  okay for manufacturing industry where management is dealing with unskilled uneducated manpower..but same is not true for Industry.Institutes still need to inculcate basic principle of psychological analysis , service industry and basic computer knowledge so people can visualize trends in skills needed to shape future of employee in company.

So For Industry still under trap of HR to define there own terminology for different job skills.In country outside india for which indian IT companies are executing project Position are well defined.Take a case Architect in software industry.

Architects can be classied into two category:

1)Enterprise Architects.

Architect working at enterprise level with various certification level like ITIL( process based for Service Management),TOGAF for Thne

Zackman for domain model, there are customised framework like NGOSS and e-TOM for Telecom industry

2) Domain specific Architects.

Domain Architect can be classied into four major types:

  1. Application or Language Architect (Java Architect,.NET Architect,PHP Architect, open source Architect, Integration Architect, Platform As Service Architect)  . These architects are primarily concerned with design patterns and application integration and language specific issues, options available like in integration web service is course grain integration but there are other option like CORBA,JNI web services is last option. So Java architect is SME for Java related design issues. same way .NET Architect, PHP Architect.Integration Architect are looking at how integrated different layer(ESB,Web services, COM,CORBA etc..).
  2. Database/Data Architect (Data Architect, ETL Architect, BI Architect, Performance Architect)
  •  Data Architect: concerned with logical data modelling (LDM) and Physical database  modelling (PDM), design of OLTP system and design normalized scehmas and table structure.
  • ETL Architect : Takes care of Data Integration layer (Extract Transform Load data from SQL server to Oracle from SAP populated tables into target datawarehouse. Designing or Star schema or extended star schema for Data warehouse.
  • BI Architect: Looks to create semantic Layer which helps user to convert dynamically dragged objects to there respective SQL equivalent.and reporting layouts ,data structure which support them,layout to of report on web on mobile or on portal.reports from tactical to strategic reporting to Exception reporting and business events.
  • Performance Architects are concerned performance to Web servers , data bases performance ,query response time, latency etc.

3. Network Architect (Storage Architect, Network Architect, System Architect,Virtualisation or Infrastructure As Service IAAS.

  • Storage Architect: Concerned with SAN storage and design of storage network its interface with network components. whether to go for SAN or NAS or DAS ,RAID levels, SCSI or Fiber channel , servers, vendors to go with (EMC Clarion,netapp,hitachi)  etc.analysts predict that enterprise storage will account for 75% of all computer hardware expenditures so storage is critical.
  • Network Architect: Network architecture have exploded in recent decade.Network architect needs to look into network Design, specification,Network security. Like Cisco defines 5 fields (1. Voice like VOIP, 2. Security, 3. Design, 4.Routing and Switching, 5.Wireless).Each one can have its own architects.person may not be specific to Cisco but can cut across to like juniper network junos firewall, or sonic firewall is in network security.
  • System Architect: concerned Types of servers used and Os Admin related activities.e,g Red hat admin subject matter expert with server datacenter know how, blade vs RACK, virtualisation layers,hypervisor (vmware ESXi, Microsoft hyperV, IBM AIX LPAR) etc.

4. Architect not defined above,

ERP Architects:

  • SAP Architect(since SAP has its own solution framework  like SAP Solution Manager  which help in maintaining and upkeep of solution and framework at solution or practice level, Also like SAP BI is domain specific architect who know out for 16 reporting tool flavor from WebI,Xcelsius,DesKI, live office, Quaas which one to use when).
  • Peoplesoft Architect: Looks into PS specific techno-functional issues customization , enhancement, or performance issue, installation and data migration, its integration with different web server app server.
  • , Oracle Apps Architect.: Oracle Application domain specific solution to use of Oracle BPEL, ESB, SOA suite, fusion middle ware vs old form report customization Vs used of OAF or ADF.