A day in life of BI Engineer part 2

Read Part1:
Part 2:
First few days should understand business otherwise cannot create effective reports.
9:00 -10am Meet customer to understands key facts which affect business.
10-12 prepare HLD High level Document containing 10,000 feet view of requirement.
version 1. it may refined later subsequent days.
12-1:30 attend scrum meeting to update status to rest of team. co-ordinate with Team Lead, Architect and project Manager for new activity assignment for new reports.
Usually person handling one domain area of business would be given that domain specific reports as during last report development resource already acquired domain knowledge.
And does not need to learn new domain..otherwise if becoming monotonous and want to move to new area. (like sales domain report for Chip manufactuers may contain demand planning etc…)
1:30-2:00 document the new reports to be worked on today.
2:00-2:30 Lunch
2:30-3:30 Look at LLD and HLD of new reports. find sources if they exist otherwise Semantic layer needs to modified.
3:30-4:00 co-ordinate with other resource reports requirement with Architect to modify semantic layer, and other reporting requirements.
4:00-5:00 Develop\code reports, conditional formatting,set scheduling option, verify data set.
5:00-5:30 Look at old defects rectify issues.(if there is separate team for defect handling then devote time on report development).
5:30-6:00 attend defect management call and present defect resolved pending issue with Testing team.
6:00-6:30 document the work done. And status of work assigned.
6:30-7:30 Look at report pending issues. Code or research work around.
7:30-8:00 report optimisation/research.
8:00=8:30 Dinner return back home.
Ofcourse has to look at bigger picture hence need to see what reports other worked on.
Then Also needed to understand ETL design , design rules/transformations used for the project. try to develop frameworks and generic report/code which can be reused.
Look at integration of these reports to ERP (SAP,peopesoft,oracle apps etc ), CMS (joomla, sharepoint), scheduling options, Cloud enablement, Ajax-fying reports web interfaces using third party library or report SDK, integration to web portals, portal creation for reports.
So these task do take time as and when they arrive.

Telecom Technology Stack

For Enterprise class large projects Architecture frameworks like TOGAF are very important:
TOGAF: The Open Group Architecture Framework was adopted and adapted to E-TOM framework or Enhanced Telecom Operational Map (E-TOM) .
E-TOM has two essential layers which cover most requirement on Telcos
{Telecom companies include (TEM) Telecom Equipment manufacturer like (ZTE, Cisco Ericsson  , Telecom Services companies like ( vodaphone, O2 etc)}.
1. Lowest Level 1: OSS (Operational System Software) which include systems which interfaces with most devices like routers, base station, towers, etc Cisco routers, Base station BTE software, NOS (network operation planning software), Activation Software, 2G, 3G stack management, tower management.
2. Upper layer: BSS (Business System Software) Which includes Business Interfacing software like Oracle Sieble CRM (Customer Relationship management ), Billing software like from Amdocs,or oracle BRMS (Billing and revenue management  software)
MIS and BI Business Intelligence software like OBIEE, SAP BO, Cognos BI etc.
Mediation: Software mediating between multiple layers.
Now lets see like BT working on E-TOM. For software implementations
British Telecom (BT) use agile methodology of keeping inventory of requirements as low as possible to keep cost down and know defects early in life cycle and then more requirements (User Stories) can be taken for development , taken to testing in SIT system integration testing and once tested OK dropped into production.
Scrum meetings are regularly held , as release happen the Burn down charts are updated telling progress status.

BT having incorporated concepts in between like
use RFT (Right First Time) 90 % of the project taken up must be completed in correct specifications in right time.
Concept to Market (C2M):Bring Concept to market faster.
Lead To Cash: (L2C)
where a lead generated to Cash in hand cycles are (Opportunity, Quotation, Order, Delivery, Issue, Invoice) L2C cycles are captured in workflow and quickened.
SAP Order Management Of Lead to Cash workflow
SAP Order Management L2CWatch Demo of SAP Lead to Cash using SAP Business By Design.

3. Trouble to Resolve (T2R). : where Each Trouble which needs resolution is trapped in CRM for reviewing the work.
4. Cycle time: Each Cycle must be pro-actively reduced in time frame.
You can Read this more detailed matrix Architecture Overview of Vision goals

Read about TOGAF:
1. Comparison of Enterprise Architecture to Domain specific Architecture Roles :
2. Enterprise Architecture and Security.
3. Links:
Oracle Seibel:
Oracle BRMS:http://www.oracle.com/us/corporate/press/2021078
Amdocs Billing software of the year:



Internet of things New Paradigm Shift in Computing

Paradigm shift in Computing Industry over period of time:

Mainframe–> Personal Computer, (PC based Application software ) –> Web Computing (Web servers, Internet, web application) –> devices (Mobile/ Mobility )/IP TV , notebook /ipad —>
For next shift there lot of possibility Like surface computing might eliminate Screen requirement or Ipad/laptop requirement, IP TV interacting with human interactions with gesture to camera , and devices projecting screen on any surface. Many devices which are coming in the industry would certainly require Ubiquitous Access. And All devices will have agent to take informed decisions (Like once fridge know milk is empty it could connect to internet and ask your access to credit card or confirmation (workflow software configured) it can order retailer.(So like Internet of Things)
So Internet of things is not only these devices that will interact with other home system, devices but also get data with wired or wireless sensors inside Home.
more about it can be read at : https://sandyclassic.wordpress.com/2013/05/03/classifying-ubiquitious-data-images-into-emotion-for-target-advertisement-campaign/

Read: https://sandyclassic.wordpress.com/2012/10/28/ubiquity-the-most-crucial-challenge-in-business-intelligence/

New age application development :

CMS are integrated with SIP servers for PSTN phone to Digital phone and Softphone conversion. More details:

All these will increase focus on the development Internet of Things with sensor network generating huge video,audio, image and text data collected from sensor has to move ubiquitous from one system to another. For this to happen internet infrastructure will be utilized using cluster computing of Hadoop, Hive, HBase. for data analysis and storage. When sensor nodes , devices , Home appliances access and interact with this data ubiquitously  at same time interact , under transaction using internet infrastructure Possibility of Internet of things is only conclusion it can derive.
Read more on hadoop: https://sandyclassic.wordpress.com/2011/10/19/hadoop-its-relation-to-new-architecture-enterprise-datawarehouse/

Relation to cloud here 3V have actually now became 5V variability and value new 2V +addition to existing 3V Volume, variety and velocity being old 3V.                                  Read more: https://sandyclassic.wordpress.com/2013/07/02/data-warehousing-business-intelligence-and-cloud-computing/

External links for reference: http://www.sap.com/index.epx
AP XI: http://help.sap.com/saphelp_nw04/helpdata/en/9b/821140d72dc442e10000000a1550b0/content.htm

Oracle Web centre: http://www.oracle.com/technetwork/middleware/webcenter/suite/overview/index.html

CMS: http://www.joomla.org/,http://www.liferay.com/http://www-03.ibm.com/software/products/us/en/filecontmana/
Hadoop: http://hadoop.apache.org/

Map reduce: http://hadoop.apache.org/docs/stable/mapred_tutorial.html
acebook API: https://developers.facebook.com/docs/reference/apis/
inkedin API: http://developer.linkedin.com/apis
witter API: https://dev.twitter.com/

Architecture and SAP Hana Vs Oracle Exadata competitive Analysis

SAP Vs Oracle
This debate of SAP Vs Oracle or last 2 yrs buzz SAP HANA vs Oracle Exalytics

Every year in Enterprise Software space Competition of SAP Vs Oracle Hots up with new announcements and New technology comparisons of SAP new Tech Vs Oracle New Tech.
The Enterprise Software stack built up by No.1 and No.2  world’s top most valued companies in Enterprise Technology Space.
So SAP Vs Oracle comparison will never go out of picture only every year it will up the ante
How Oracle Exadata and SAP HANA similar?

First, they are both are appliances – bundles of hardware and software and services that give you something that will run out the box. This means that the 3 elements: hardware, software and tuning – are highly optimised to work together.

Second, they are both initially optimised for Data Warehousing scenarios.

Third, both scale more or less linearly for large datasets.

SAP HANA more detailed:

Databases as we see earlier evolved from file based systems to Row oriented (RDBMS ) databases. to Column Oriented databases like SAP HANA (http://blogs.sap.com/SAPHANA/),that stores its content by column rather than by row.This has advantages for data warehouses and library catalogues where aggregates are computed over large numbers of similar data items. But what about OLTP Systems we are told that remove layers and everything in memory and it is not disruptive.But Still there are probabilitic databases which are advanced then column oriented databases.but are under evolution..

SAP HANA Architecture: why is distruptive? what problem does it solves?:

Normal Web App architerture : Portal linked to ESB which is taking to multiple Web Apps(these Apps have there own databases).Then We have ETL taking all these delta loads to EDW.Ontop of EDW we have multiple Analytical Apps. There are 3 problem in this design:

1. Problem #1: There is always Delta between App and EDW.Also Delta between App.

2.Problem #2: Since Delta between App and Analytic App decisions are delayed or misrepresented by delta leading delay in Actions.

3. Problem #3: Value in App T1 > value in Analytic App T2.

Solution presented By HANA:

1. For All events happening in databases like any changes should trigger event . Called Complex Event Processing CEP. It will collect and process all events.These changes also needs to replicated quickly to other associated events databases..hence Sybase Replication Server comes handy.

2. Replace Application databases and EDW databases with Single in memory DB (inside RAM).And One Archival database.All App and Analytical App point to In-Memory DB hence there is no Delta between them.With these 2 changes above problems are solved by HANA.

Some time back , Bob Evans from SAP wrote a blog entitled The Top 10 Reasons SAP HANA Is Disrupting Larry Ellison’s Grand Plans.

Link: http://blogs.sap.com/bobevans/2011/06/23/the-top-10-reasons-sap-hana-is-disrupting-larry-ellison%E2%80%99s-grand-plans/

How are Oracle Exadata and SAP HANA fundamentally different?

Oracle Exadata is a cell based Oracle appliance. This means that an Exadata Appliance is made up of multiple Oracle Exadata Storage Servers, each of which handles a workload, and part of a response set. So you ask a question of Exadata and the central node chunks stuff up and the Storage Servers crunch out responses. The node compiles those responses into an overall response.

SAP HANA is an in-Memory Appliance. It does break up responses if you have multiple nodes just like Exadata, but that’s not really the point. The point is that even with a single HANA node, you can get blistering performance from a reporting query, because it performs the calculation in main memory.

HANA has an application layer that sits with the In-Memory Database. This is the killer blow because the first native app is a calculation engine. This means it can do complex calculations without an application layer, which massively improves application performance.

So the key here is that whilst Exadata remains a database, SAP are pushing HANA as an application platform – and with good reason.

SAP new version come with (netweaver,BI,ABAP,Java). Enterprise workspaces into portal.

SAP improvements : In 2011 we had following changes:

Stack Vs Non Stack: War of Words

Semantic layer can handle (universes,dimensional data and multi-dimensional data, integrated to netweaver (people integration,data integration, process integration)(includes identity/lifecycle management) event processing, 16 clients report  flavour together into single design time.But is not netweaver developed on Java open source platform.But going forward its with Oracle.Is SAP going to pay royality on everything it create over it?  Where do we go from here?. History: SAP worked withIBM  to put away its default databases on vanilla install from Oracle to IBM DB2. then came new installer for vanilla install IBM+SAP products. Is is not stackized approach? Still it not how open source or collaborative source software are based.

2. Then SAP long back came with announcement that we are doing away with dependency of Java..and came BSP to take over at-least JSP.but where does it stands about open Systems like Java eco-systems where we have community system.Does Any SAP tech has ecosystem. If there is no eco-system will system evolve..that’s where Oracle has taken leap faith.A  community can create and breed new technology or a closed wall announcements.My take is SAP customers are more traditional enterprises that have long relationships with the existing ERP platform and appreciate stable upgrade paths, maintenance and support to keep operations running smoothly.

3.HANA: HANA(http://blogs.sap.com/SAPHANA/) can process both structured , unstructured, machine generated and also Social networking  data. So is Hadoop which uses the strategy of bringing the computation to the data instead of transferring the data for computation thus reducing the network delay and further running Map reduce algorithms over the data. (http://developer.yahoo.com/hadoop/ )Hadoop doesn’t suit for interactive data processing.SAP HANA focuses at interactive or realtime analytics whereas Big Data processing technologies like Hadoop which is an open source and deals with petabytes and exabytes and is not realtime based.Mere realtime examples are analyzing the logs generated by web servers of enterprises and Facebook or Linked-In data which generates hundreds of terabytes data across world everyday. But this can be analyzed with any other data warehousing tool.

CRM: Customers will be interested in running data analytics against increasing amounts of data stored in both SAP and non-SAP systems. Hence mostly they do not employ researchers or data scientists to experiment with still developing technologies like Hadoop for distributed computing and Big Data analytics.

Finally according to me HANA is not about Bigdata but more about quick or fast data for  all its customers to make quicker, better business decisions to stay ahead of the competition and hence puts SAP in a good position to expand its analytics business.But we will look further is it really able to reduce layers.

Oracle Exadata:SAP certified for exadata:



Important Question While considering SAP HANA:

1) Cost benefit analysis :Whilst true the cost of hardware (memory / CPU) keeps decreasing and how much do you think you will be spending to deliver a system 3600x faster than your current one? Its the usual cost to benefit comparison.

2) HANA is underpinned by solid state storage / this can be raided etc for redundancy. HANA offers snapshot backups for recovery / HA / DR purposes. In its current version although it has a log capability (similar to transaction log/archive log), this log does not offer a backup option so “point in time recovery” is only available locally. ie in a disaster you could only recover to the most recent snapshot. It is for this reason (and probably others) that SAP is only focussing on it’s use in warehouse type scenarios (eg BW7.3x is to run on HANA) where it is not the database of record. In the upcoming versions when some of these enterprise database facilities are deployed then SAP I’m guessing will offer HANA as the DB for it’s operational applications (ERP / CRM etc).

4. Sybase: Mission critical financial application and mobility.Tech talks about stack-less future of heterogeneity.

-Hana comes with columar database not the latest probalistic databases.

Watch youngest india head ever presenting SAP Teched in india.

SAP new challenge.: cloud and the legacy of ABAP…what it means in BI..let see it..

SAP chairman said some last tech ed in europe..next 5 years SAP needs to work what it has not last 20 years…What is it? we see try understand that why? short 1. ABAP is old language still long way to go when compared with Java when it used for Fusion on which Oracle ERP is based.. SAP Answers: 1 Develop: BSP business Server pages to take on JSP in Java.. remove dependency on Java hence on netweaver..Where should netweaver go.. 2. Competition from cloud: SAP knows it limitation hence tied up best cloud provider Google..BO analytics. 3.new virtualization technology and parallelism with intel.

Few more things happen as processor hit More’s law we had to increase processing power using parallelism, intel came with new C++ API to increase parallelism of software (http://software.intel.com/en-us/articles/intel-parallel-studio-xe/).Then SAP came up with HANA in partnership with intel. see more details below..

Just see few changes on business Objects due to cloud and new mesh up architecture popularity..now BO is using google API to present it over mesh up or map API.Open system of technology ecosystems for SAP ISV, technology partners to customize and present to SAP systems.


It gives google good entry point in enterprise software. So do  we see google going deep into it..Even for mobilty SAP acquired Sybase but still mobility google can provide better solution.Now SAP has two choices Sybase or google API for Mobility.

Consumption strategy to increase usage : light components like Adobe flex for outside system/people interact with data and functionality of System and integrate with gateway to SAP for consumptions,front-end technology like adobe flex which gives access to outside systems,or in premise systems like Ms office,sharepoint and mobilty application came with acquisition of sybase.

Cloud strategy of SAP: product strategy is to provide systems On premise, on demand, on device. consumption like webservices,cloud for virtualization and in-memory for quick response time.Cloud systems have three Type SAAS,PAAS,IAAS

SAAS/PAAS  Strategy: Platform as service: Platforms: 1 SAP by Design Platform: core application, transactional,robust enterprise class application(application like sales on demand)

SAP Business ByDesign which is ideally suited for small and medium size companies and subsidiaries of large corporations.Re-Align your modules priority quickly..by using design..build from ground up.

watch demo video:  http://www.sap.com/solutions/products/sap-bydesign/index.epx

2.River Platform: River is a cloud-based development environment which supports lightweight extensions to on-premises ERP (enterprise resource planning) software.It is based on SAP’s purchase of Coghead in 2009  is present in the urns in the REST API.

application have rapid application development,collaborative applications.Carbon impact applications.http://www.sapcarbonimpact.com/

Both have backed integration capability with on premise  systems.

IAAS: Infrastructure as service VCM: External cloud and Internal cloud, virtualized machines all can be managed with virtualization and cloud management to make IT more Agile and move from like development systems to production systems quickly. self service portals for maintaining cloud. (VCM landscape management software (BC-VCM-LMS)

Partnership with Intel: to increase parallisation to utilize hardware to increase compute capacity.In memory computing engine.

SAP HANA in memory unifying architecture is path breaking but where it can be leveraged ? What HANA does is integrate

High performance Analytical appliance. HANA in memory computing engine on which partner can build appliance like IBM,HP,Fijistsu,Dell. It can give sub second response time. modelling and data model from BO, data replication through sybase transnational systems into this in memory database  .

Three protocol:1.  BICS for BO, 2. SQL for other applications on top of it and 3.MDX for applications like Excel.

HANA can power any mobile applications and gateway to easy consumption and attach to existing system.Exiting customer can also utilize and hence non disruptive.can utilize latest consumption models and infrastructure innovations.Lower TCO.

Enterprise Architecture TOGAF,ITIL,Zachman,eTom,NGOSS


– Business Frameworks (e.g. BMM, Six Markets, Porters 5 Forces, McKinsey 7S, etc)
– Architecture Frameworks (e.g. PEAF, MODAF, TOGAF, TEAF, IAF, etc)
– Programming Frameworks (e.g. .NET, J2EE, etc)
– Project Management Frameworks (e.g. PRINCE2, MSP, etc)
– Service Management Frameworks (e.g. ITIL, etc)
– Industry Operations Frameworks (e.g. eTOM, Pragmatic Marketing, etc)
– Any other type of framework in any other domain so long as it is related to an ENTERPRISE.

There are various field of Architecture floating around in IT industry.Specially in indian IT industry where things are still taking shape.HR department is india in most forgotten department it still need to learn more about Labour laws and nothing more..Top institution in india for HR which was essentially focus on labour laws and nothing more.Other subjects are taught not with great emphassis..Problem that is  okay for manufacturing industry where management is dealing with unskilled uneducated manpower..but same is not true for Industry.Institutes still need to inculcate basic principle of psychological analysis , service industry and basic computer knowledge so people can visualize trends in skills needed to shape future of employee in company.

So For Industry still under trap of HR to define there own terminology for different job skills.In country outside india for which indian IT companies are executing project Position are well defined.Take a case Architect in software industry.

Architects can be classied into two category:

1)Enterprise Architects.

Architect working at enterprise level with various certification level like ITIL( process based for Service Management),TOGAF for Thne

Zackman for domain model, there are customised framework like NGOSS and e-TOM for Telecom industry

2) Domain specific Architects.

Domain Architect can be classied into four major types:

  1. Application or Language Architect (Java Architect,.NET Architect,PHP Architect, open source Architect, Integration Architect, Platform As Service Architect)  . These architects are primarily concerned with design patterns and application integration and language specific issues, options available like in integration web service is course grain integration but there are other option like CORBA,JNI web services is last option. So Java architect is SME for Java related design issues. same way .NET Architect, PHP Architect.Integration Architect are looking at how integrated different layer(ESB,Web services, COM,CORBA etc..).
  2. Database/Data Architect (Data Architect, ETL Architect, BI Architect, Performance Architect)
  •  Data Architect: concerned with logical data modelling (LDM) and Physical database  modelling (PDM), design of OLTP system and design normalized scehmas and table structure.
  • ETL Architect : Takes care of Data Integration layer (Extract Transform Load data from SQL server to Oracle from SAP populated tables into target datawarehouse. Designing or Star schema or extended star schema for Data warehouse.
  • BI Architect: Looks to create semantic Layer which helps user to convert dynamically dragged objects to there respective SQL equivalent.and reporting layouts ,data structure which support them,layout to of report on web on mobile or on portal.reports from tactical to strategic reporting to Exception reporting and business events.
  • Performance Architects are concerned performance to Web servers , data bases performance ,query response time, latency etc.

3. Network Architect (Storage Architect, Network Architect, System Architect,Virtualisation or Infrastructure As Service IAAS.

  • Storage Architect: Concerned with SAN storage and design of storage network its interface with network components. whether to go for SAN or NAS or DAS ,RAID levels, SCSI or Fiber channel , servers, vendors to go with (EMC Clarion,netapp,hitachi)  etc.analysts predict that enterprise storage will account for 75% of all computer hardware expenditures so storage is critical.
  • Network Architect: Network architecture have exploded in recent decade.Network architect needs to look into network Design, specification,Network security. Like Cisco defines 5 fields (1. Voice like VOIP, 2. Security, 3. Design, 4.Routing and Switching, 5.Wireless).Each one can have its own architects.person may not be specific to Cisco but can cut across to like juniper network junos firewall, or sonic firewall is in network security.
  • System Architect: concerned Types of servers used and Os Admin related activities.e,g Red hat admin subject matter expert with server datacenter know how, blade vs RACK, virtualisation layers,hypervisor (vmware ESXi, Microsoft hyperV, IBM AIX LPAR) etc.

4. Architect not defined above,

ERP Architects:

  • SAP Architect(since SAP has its own solution framework  like SAP Solution Manager  which help in maintaining and upkeep of solution and framework at solution or practice level, Also like SAP BI is domain specific architect who know out for 16 reporting tool flavor from WebI,Xcelsius,DesKI, live office, Quaas which one to use when).
  • Peoplesoft Architect: Looks into PS specific techno-functional issues customization , enhancement, or performance issue, installation and data migration, its integration with different web server app server.
  • , Oracle Apps Architect.: Oracle Application domain specific solution to use of Oracle BPEL, ESB, SOA suite, fusion middle ware vs old form report customization Vs used of OAF or ADF.